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software  packages  are  released,  the  results  of  this  study  may  change. 
However,  the  methodology  developed  in  this  thesis  may  still  be  applied  in 
studies  conducted  in  the  future. 

There  are  several  people  whom  I  wish  to  thank  for  their  assistance 
in  completing  this  study.  First  is  my  thesis  advisor,  Capt  David 
Christensen,  for  his  advice  and  guidance  in  keeping  the  study  focused  on 
the  primary  objective  of  the  thesis.  I  would  also  like  to  thank  the 
focal  points  for  all  software  packages  that  were  used  and/or  evaluated  in 
this  study.  Their  assistance  in  obtaining  and  evaluating  the  software 
packages  is  greatly  appreciated.  Finally,  I  would  like  to  thank  my 
family  and  friends  for  their  understanding  and  support.  Most  of  all, 
however,  I  wish  to  thank  my  family  and  especially  my  parents,  James  and 
Bernice  (alias  Mom  and  Dad),  for  their  prayers  and  support  not  only 
during  this  time  but  always . 


Terry  V.  Smith 


Table  of  Contents 


Page 

Preface .  ii 

List  of  Tables . v 

Abstract .  vi 

I .  Introduction  .  1 

General  Issue  ...  .  1 

Specific  Problem  . 1 

Investigative  Questions  .  2 

Limitations  .  2 

Assumptions .  3 

Road  Map .  3 

II.  Literature  Review  .  5 

Overview  .  5 

Justification  of  the  Literature  Search  .  5 

Scope  of  the  Research  Topic  .  7 

Discussion  of  the  Literature  .  7 

Background  .  7 

What  is  a  CPR  versus  a  C/SSR  .  10 

How  to  Evaluate  the  CPR .  11 

CPR  Software  Analysis  Packages  .  17 

Criteria  for  Selecting  Software  Packages  .  18 

Project  Management  Study  .  19 

AFSC  Software  Quality  Indicators  .  24 

Operations  Research  Study  .  24 

Word  Processor  Study .  26 

Summary  of  the  Literature  .  28 

III.  Methodology  .  31 

Phase  I:  Initial  Selection  of  Criteria  to  Evaluate 

CPR  PC  Software .  32 

Phase  II:  Analysis  of  Criteria  for  Software 

(Criteria  Validation  by  Users)  .  33 

Phase  III:  Final  Selection  of  Evaluation  Criteria  .  34 

Phase  IV:  Method  of  Evaluation .  44 

Phase  V:  Software  Evaluation  and  Comparison  Using 

Criteria  Established .  48 

Performance  Analyzer  (PA)  .  48 

CPR-EZ .  55 

Contract  Appraisal  System  (CAPPS)  ....  62 

Contract  Cost  Analysis  System  (CCAS).  .  .  73 


iii 


Page 


IV.  Results,  Conclusions  and  Reconmendations  .  85 

Selection  of  "the  Ideal"  CPR  Software  Package  or  Features 

of  "the  Ideal"  Software  Package  .  85 

Expert  Choice  Verbal  Node .  85 

Sensitivity  Analysis  of  Verbal  Mode  .  86 

Expert  Choice  Numerical  Mode .  89 

Sensitivity  Analysis  of  Numerical  Mode .  91 

Conclusions  and  Reconmendations  from  Comparing  the 
Results  of  the  Verbal  and  Numerical  Modes  of 

Expert  Choice .  92 

Further  Study  .  93 

Appendix  A:  CPR  Analysis  Software  Packages  .  95 

Appendix  B:  Detailed  Analysis  of  Selecting  "the  Ideal"  CPR  Analysis 

Software  Package  Using  Expert  Choice  .  101 

Bibliography  .  117 

Vita . 119 


xv 


List  of  Tables 


Table  Page 

1.  Summary  of  Software  Criteria  for  Four  Different 

Studies .  29 

2.  First  Comparison  Method  .  35 

3.  Second  Comparison  Method  .  37 

4.  Third  Comparison  Method  .  38 

5.  Summary  of  the  Three  Comparison  Method  Results  .  39 

6.  Comparison  of  CPR  Software  Criteria  to  Software 

Criteria  for  Four  Different  Studies  .  40 

7.  Rankings  of  "Most  Important"  CPR  Software  Criteria 

Determined  from  Three  Comparison  Methods  .  42 

8.  Order  of  Importance  of  "Most  Important"  CPR  Software 

Criteria . 44 

9.  Comparison  of  Performance  Measurement  Information  in 

CAPPS  User's  Manual  vs  Computer  Software  .  71 

10.  SuniLary  for  Evaluation  of  CPR  Analysis  Software  Programs 

Against  User-Established  Criteria  .  84 

11.  Rankings  of  Four  CPR  Analysis  Software  Programs  Using 

the  Expert  Choice  Verbal  Mode .  86 

12.  Weights  for  the  General  Criteria  and  Subcriteria  Categories 

in  Conducting  a  Numerical  Analysis  of  the  CPR  Software 
Packages .  90 

13.  Rankings  of  Four  CPR  Analysis  Software  Programs  Using 

the  Expert  Choice  Numerical  Mode .  90 


v 


AFIT/GCA/LSY/90S-8 


Abstract 


n  / 


J 


r\ 

\jS 


The  purpose  of  this  «*«dy  was  to  determine  the  ideal"  personal 
computer  (PC)-based  Cost  Performance  Report  (CPR)  analysis  software 
package  currently  available.  In  conducting  this  study,  evaluation 
criteria  needed  to  be  selected  for  analyzing  the  software.  In  order  to 
establish  what  criteria  were  to  be  used  in  evaluating  the  CPR  analysis 
software,  two  questionnaires  were  sent  to  Air  Force  Systems  Command 
(AESCV  product  divisions.  The  first  questionnaire  asked  the  users  what 
features  would  be  useful  and/or  ideal  in  a  CPR  analysis  software  package. 
The  second  questionnaire  then  asked  the  users  to  rank  order  the  criteria 
established  from  the  first  questionnaire.  The  final  evaluation  criteria 
were  then  selected  based  on  an  analysis  of  the  responses  to  the  second 
questionnaire.  Upon  selection  of  the  criteria,  four  PC-based  CPR 
analysis  software  packages  were  evaluated  against  those  criteria. 

Further  analysis  of  the  CPR  software  packages  was  conducted  on  a 
decision  support  software  package.  Expert  Choice.  Expert  Choice  ranked 
the  CPR  software  packages  with  respect  to  the  evaluation  criteria 
established.  A  sensitivity  analysis  was  also  performed  with  regard  to 
the  importance  of  the  evaluation  criteria  in  ranking  the  software 


packages . 


CPR  SOFTWARE  ANALYSIS : 
DETERMINING  "THE  IDEAL"  SOFTWARE 


I.  Introduction 


General  Issue 

The  Department  of  Defense  must  obtain  performance  data  on  contracts 
established  with  contractors  in  order  to  determine  how  those  contractors 
are  performing.  To  monitor  cost  and  schedule  performance,  the  government 
requires  contractors  to  submit  various  performance  reports.  The  primary 
report  required  for  doing  this  is  the  Cost  Performance  Report  (CPR).  The 
CPR  reports  cost  and  schedule  progress  by  comparing  "planned  work  versus 
actual  work  performed  and  actual  work  performed  versus  actual  costs 
expended"  (2:H-11).  In  order  to  analyze  contractor  pe 'f ormance ,  many 
software  programs  have  been  developed  to  evaluate  the  data  on  CPRs . 

Specific  Problem 

From  the  many  different  software  programs  that  are  available  to 
analyze  contractor  data,  no  one  standardized  program  has  been  developed 
or  approved  for  use  within  the  Department  of  Defense.  The  issues  to  be 
considered  are  not  only  which  of  these  software  programs  is  the  best 
evaluator  of  CPR  data,  but  is  there  "an  ideal"  program  from  the  ones  that 
are  currently  available?  "Ideal,"  as  described  here,  will  be  determined 
from  criteria  established  by  users  of  CPR  software  in  Air  Force  Systems 
Command  (AFSC)  product  divisions.  If  there  is  not  "an  ideal"  program 
currently  available,  what  combination  of  features  from  certain  software 
packages  would  result  in  "the  ideal"  evaluation  of  CPR  data? 
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Investigative  Questions 

1.  What  software  packages  are  currently  available  in  both  the  government 
and  from  contractors  for  evaluating  CPR  data? 

2.  How  do  government  and  contractor  packages  compare  in  terms  of 
similarities  and  differences?  How  do  the  software  packages  compare  in 
terms  of  speed  or  processing  time?  Are  they  "user  friendly”? 

3.  What  criteria  are  needed  to  evaluate  CPR  analysis  software?  Are 
current  software  packages  mathematically  accurate  and  do  the  different 
packages  provide  the  same  result  for  the  operation  being  performed — in 
calculating  the  cumulative  Budgeted  Cost  of  Work  Performed  (BCWP), 
Budgeted  Cost  of  Work  Scheduled  (BCWS),  Actual  Cost  of  Work  Performed 
(ACWP),  Cost  Variance  (CV)  and  Schedule  Variance  (SV)? 

4.  Are  the  criteria  established  in  (3)  above  met  in  the  software 
packages  already  available? 

5.  What  information/features  are  omitted  from  current  software  packages 
that  government  evaluators  think  should  be  included?  Is  there  a  software 
package  currently  available  that  is  better  than  the  others  for  evaluating 
CPR  data?  If  not,  what  combinations  of  features  from  different  packages 
would  result  in  "the  ideal"  software  package?  And  is  there  any  other 
information  and/or  features  that  need  to  be  included  that  are  not 
currently  included  in  the  software  packages? 

Limitations 

The  Samsung  S330  personal  computer  (PC)  system  with  an  MS-DOS 
version  3.30  operating  system  and  a  Panasonic  dot  matrix  printer  was  used 
to  conduct  this  research.  Because  the  Samsung  S330  is  an  IBM-compatible 
machine,  this  study  is  designed  to  analyze  software  that  is  compatible 
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with  the  IBM  personal  computer.  Because  the  IBM-compatible  personal 
computer  is  the  type  of  technology  currently  being  used  in  the  Air  Force 
at  the  time  of  this  study,  it  was  chosen  as  the  basis  for  this  study. 
Therefore,  this  study  will  evaluate  PC-based  CPR  analysis  software  only 
and  will  not  include  mainframe  CPR  analysis  software. 

Assumptions 

The  most  important  assumption  underlying  this  study  is  that  once  the 
criteria  for  CPR  software  analysis  have  been  identified  and  verified,  a 
standardized  software  package  can  be  developed.  This  software  package 
could  not  only  result  in  standardized  procedures  for  condu  ting  CPR 
analysis  but  may  also  result  in  reduced  expenditures  on  software  packages 
by  the  DOD.  Because  software  packages  are  continually  being  updated,  it 
is  assumed  that  the  criteria  decided  upon  will  be  able  to  be  used  in 
selecting  CPR  software  packages  in  future  years. 

Road  Map 

In  answering  the  above  investigative  questions,  the  following  steps 
will  be  taken.  First  of  all,  Chapter  II  provides  background  information 
on  the  Cost  Performance  Report  (CPR),  lists  the  names  of  the  four  CPR 
software  packages  that  will  be  evaluated  in  this  study  and  provides 
information  on  criteria  used  in  four  different  studies  for  selecting 
software  packages  for  various  types  of  functions  (i.e.,  project 
management  and  operations  research) . 

As  indicated  above,  evaluation  criteria  will  need  to  be  selected  for 
analyzing  the  CPR  software.  In  order  to  establish  what  criteria  should 
be  used  to  evaluate  CPR  analysis  software,  two  questionnaires  will  be 
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sent  to  Air  Force  Systems  Command  (AFSC)  product  divisions.  The  first 
questionnaire  will  ask  the  users  what  features  would  be  useful  and/or 
ideal  in  a  CPR  analysis  software  package.  The  second  questionnaire  will 
then  ask  the  users  to  rank  order  the  criteria  established  from  the  first 
questionnaire.  The  final  evaluation  criteria  will  then  be  established 
based  on  analysis  of  the  second  questionnaire.  After  the  criteria  have 
been  selected,  the  four  CPR  analysis  software  packages  will  be  evaluated 
against  these  criteria.  Chapter  III,  Methodology,  will  consist  of  five 
phases  in  which  the  above  analysis  will  be  performed. 

Chapter  IV,  Results,  Conclusions  and  Recommendations,  will  consist 
of  further  analysis  of  the  four  CPR  software  packages  on  a  decision 
support  software  package.  Expert  Choice.  Expert  Choice  will  rank  the  CPR 
software  packages  with  respect  to  the  evaluation  criteria  established  in 
Chapter  III.  A  sensitivity  analysis  will  also  be  performed  with  regard 
to  the  importance  of  the  evaluation  criteria  in  ranking  the  software 
packages . 
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II.  Literature  Review 


Overview 

The  purpose  of  this  literature  review  is  threefold.  First,  the 
basic  concepts  of  the  Cost  Performance  Report  (CPR)  will  be  discussed. 
This  will  include  the  definition  of  what  a  CPR  is  and  how  it  is  used  to 
evaluate  contractor  data.  Second,  a  list  and  description  of  the 
potential  CPR  software  packages  that  will  be  analyzed  will  be  discussed. 
Third,  generic  criteria  used  in  selecting  software  packages  for  functions 
other  than  CPR  analysis  will  be  reviewed.  This  review  will  provide  an 
introductory  knowledge  base  for  the  methodology  chosen  in  conducting  the 
research. 

Justification  of  the  Literature  Search 

The  justification  of  this  research  is  based  on  the  fact  that  time 
and  money  have  been  spent  needlessly  by  different  companies  who 
regenerate  computer  software  programs  that  basically  perform  the  same 
functions.  This  further  results  in  confusion  as  to  which  software 
program  is  the  best  for  performing  a  particular  function  and  should  be 
used  in  the  DOD.  When  separate  commands  or  separate  product  divisions 
within  Air  Force  Systems  Coninand  (AFSC)  buy  different  software  packages 
to  perform  the  same  function,  the  result  may  be  nonstandardization  of 
performance  evaluation  throughout  AFSC  and  the  DOD.  It  is  difficult  to 
determine  problem  areas  in  the  data  when  using  nonstandardized 
techniques.  When  using  these  techniques,  the  product  divisions  may  use 
one  type  of  software  to  conduct  the  work  while  the  headquarters  section 
may  use  another  type  of  software  to  evaluate  the  work  performed  on  the 
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same  data.  This  may  create  potential  inconsistencies  in  evaluating  the 
data . 

Buying  one  software  program  to  perform  a  particular  function  not 
only  results  in  reduced  confusion  as  to  how  to  evaluate  the  data  but  may 
also  result  in  increased  savings.  The  economies  from  buying  in  large 
quantities  are  well-known  and  the  government  could  realize  substantial 
savings  from  buying  one  software  package  for  all  of  DOD  from  one 
contractor.  However,  since  the  topic  of  single-source  versus  multiple- 
source  contracting  is  not  the  subject  of  this  research  topic,  the 
advantages  and  disadvantages  of  each  will  not  be  discussed. 

In  determining  which  software  package  to  purchase  for  performing  a 
particular  function,  the  first  objective  that  needs  to  be  met  is 
establishing  the  criteria  for  selecting  that  software.  The  personnel 
that  need  to  be  involved  in  selecting  the  criteria  are  the  users  of  the 
potential  software  and  the  evaluators  of  the  work  that  the  users  perform. 

After  the  criteria  are  determined,  the  developers  of  the  software, 
along  with  the  users  and  the  evaluators,  need  to  determine  what  features 
should  be  included  in  the  software  in  order  to  meet  the  criteria  chosen. 
They  also  need  to  discuss  how  those  features  can  be  constructed  so  that 
the  user  is  able  to  understand  how  to  use  the  software  in  performing  his 
or  her  work.  In  other  words,  the  software  needs  to  be  "user  friendly." 
However,  a  necessary  part  in  determining  what  the  ideal  CPR  software 
package  should  include  will  involve  more  than  soliciting  criteria  from 
users  of  the  software.  Comparing  the  currently  available  software  to 
these  criteria  is  necessary  in  order  to  determine  "the  ideal"  CPR 
analysis  software  package  available  at  this  time. 
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Scope  of  the  Research  Topic 


There  are  basically  two  purposes  for  the  research  that  will  be 
conducted.  First,  the  purpose  of  this  study  is  to  determine  the  criteria 
necessary  to  evaluate  the  software  used  in  conducting  CPR  analysis. 

(Some  examples  of  these  criteria  may  include  the  following:  user 
friendliness  -  easy  to  use  or  can  load  the  software  onto  the  computer 
with  no  problems;  documentation  -  easy  to  understand  and  follow,  indexes 
all  features;  tutorials  -  explains  all  features  and  includes  "hands  on" 
examples;  help  menus  -  includes  additional  "help"  that  thoroughly 
explains  how  to  solve  the  problem;  easy  to  learn  -  takes  approximately 
one  hour  to  learn  how  to  use  the  software.)  Once  the  criteria  are 
selected,  software  packages  will  be  evaluated  against  these  criteria. 

Then  either  "the  ideal"  software  package  will  be  selected  or  a  list  of  the 
features  to  be  included  in  "the  ideal"  software  package  will  be  deter¬ 
mined.  Criteria,  or  features,  are  the  desirable  characteristics  included 
in  CPR  analysis  software  for  evaluating  CPR  data  submitted  by  contractors 
to  the  government.  Therefore,  the  following  terms  will  be  used  inter¬ 
changeably:  criteria,  features  and  characteristics.  Second,  because 

IBM-compatible  computers  are  widely  used  in  the  Air  Force,  personal 
computer  (PC)  software  programs  will  be  evaluated  in  this  study  versus 
programs  conducted  with  mainframe  computers. 

Discussion  of  the  Literature 

Background .  In  order  to  determine  how  government  contractors  are 
performing  on  various  contracts  that  have  been  established  with  the 
Department  of  Defense  (DOD),  the  government  must  obtain  performance  data 
on  the  contracts  from  the  contractors'  Management  Control  System  (MCS). 
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A  Management  Control  System  is  a  system  "used  by  a  contractor  to  plan  and 
to  control  the  cost  and  scheduling  of  work  .  .  .  [and  includes]  planning, 
scheduling,  budgeting,  estimating,  work  authorization,  cost  accumulation, 
performance  measuring,  and  so  on"  (7:513).  The  quality  and  standard¬ 
ization  of  the  MCS  determine  whether  or  not  this  data  can  be  relied 
upon.  To  ensure  that  contractor  data  are  reliable,  the  contractor's  MCS 
must  be  "validated  as  being  acceptable  under  the  Cost/Schedule  Control 
System  Criteria  (C/SCSC)"  (2:5-25).  AFR  800-6,  AFSC  Supplement  1  (Ref. 
[8]),  Attachment  4  contains  guidance  on  how  to  apply  C/SCSC.  It  also 
"specifies  any  cost  reports  that  are  required  by  type  of  contract  and 
dollar  value"  (2:5-25). 

To  monitor  cost  and  schedule  performance,  the  government  requires 
contractors  to  submit  various  performance  reports.  The  primary  report 
required  for  doing  this  is  the  Cost  Performance  Report  (CPR).  When 
working  with  the  Air  Force,  contractors  are  required  to  submit  CPRs  each 
month  for  "R&D  programs  over  $40  million  or  $160  million  production 
contracts.  .  .  .  FFP  [firm-fixed  price]  contracts  are  [usually]  excluded 
from  CPR  requirements"  (2:H-11).  FFP's  are  usually  excluded  because  the 
contractor  assumes  all  of  the  cost  risk  in  a  FFP  contract.  The  CPR 
reports  cost  and  schedule  progress  by  comparing  "planned  work  versus 
actual  work  performed  and  actual  work  performed  versus  actual  costs 
expended"  (2:H-11).  For  contracts  "that  have  lower  dollar  values  than 
those  requiring  C/SCSC,  a  C/SSR  [Cost/Schedule  Status  Report]  is 
required”  (2:5-26).  Currently,  a  C/SSR  is  required  for  contracts  over  $2 
million  which  have  contract  periods  greater  than  twelve  months  and  do  not 
require  a  CPR  (7:239).  However,  page  298  of  the  draft  version  of 
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Department  of  Defense  Directive  (DODD)  5000.1,  dated  1989,  states  that 
the  $2  million  requirement  will  be  changed  to  $5  million  in  1989  constant 
dollars . 

The  basic  intent  of  the  CPR  is  to  provide  both  the  contractor  and 
the  Government  with  information  regarding  the  performance  of  the 
contractor  with  respect  to  cost  and  schedule.  DODI  7000.2,  Performance 
Measurement  for  Selected  Acquisitions,  requires  the  contractor  to 
furnish  this  information  to  the  Government  when  certain  criteria  (R&D 
programs  over  $40M  and  production  contracts  over  $160M)  for  a  contract 
have  been  met.  The  CPR  is  a  valuable  management  tool.  It  not  only 
provides  reliable  and  structured  cost  and  schedule  information,  but  this 
information  is  used  to  monitor  and  evaluate  the  performance  of  the 
contractor.  CPR  information  can  be  used  to  identify  cost  and  schedule 
problems  early  in  the  program  as  well  as  determine  the  impact  or  the  size 
of  potential  problems  in  those  areas.  The  CPR  is  typically  used  by  the 
contractor  in  addition  to  other  program  management  techniques.  If  the 
contractor  is  unable  to  consistently  track  performance  to  the  CPR,  this 
may  be  an  indication  to  the  contractor  and  the  Government  that  something 
is  wrong.  An  in-depth  analysis  should  then  be  conducted  to  determine  the 
cause  of  the  problem.  The  CPR  is  also  used  to  report  to  Congress  when 
contractors  break  the  15%  or  25%  thresholds  (7:239).  Variance  thresholds 
are  outer  limits  for  cost  and  schedule  parameters  and  "when  a  performance 
threshold  has  been  penetrated  beyond  a  previously  agreed-to  value,  a  sort 
of  'buzzer'  goes  off,  and  certain  actions  must  take  place  in  order  to 
comply  with  the  rules  of  C/SCSC"  (7:106). 
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What  is  a  CPR  versus  a  C/SSR.  The  CPR  and  C/SSR  contain  important 
differences  other  than  the  dollar  threshold  amounts.  This  can  be 
illustrated  as  follows.* 

Cost  Performance  Report  (CPR').  The  purpose  of  the  CPR  is  to 
allow  the  contractor  to  identify  variances  in  cost  and  schedule 
performance  from  the  budgeted  and  scheduled  work  that  was  originally 
planned.  The  CPR  enables  the  contractor  to  analyze  in  detail  the 
financial  condition  of  the  contract  by  being  able  to  relate  problems  in 
terms  of  their  dollar  value  and  by  displaying  any  trends  that  may  be 
developing.  The  CPR  is  composed  of  the  following  five  formats: 

Format  1  (Work  Breakdown  Structure  or  WBS):  Data  are  provided  by 
summary  level  WBS  structure  elements  to  evaluate  cost  and 
schedule  performance. 

Format  2  (Functional  Categories):  Data  are  provided  by 

organizational  or  functional  cost  categories  to  evaluate 
cost  and  schedule  performance. 

Format  3  (Baseline):  Performance  is  measured  against  the  budget 
baseline . 

Format  4  (Manpower  Loading):  Forecasts  of  manpower  loading 

illustrates  the  relationship  of  manhours  with  the  budget 
plan  and  estimated  cost. 

Format  5  (Problem  Analysis):  An  explanation  of  significant  contract 
problems  including  significant  cost  and  schedule  variance 
problems  is  included  in  this  report  written  by  the 
contractor  (13:2-3). 

Cost/Schedule  Status  Report  (C/SSR).  The  C/SSR  is  used  to 
report  C/SCSC  information  on  programs  "too  small  to  qualify  for  full 
C/SCSC  application"  (13:3).  Whereas  the  CPR  consists  of  five  formats  and 
requires  some  depth  of  reporting,  the  C/SSR  consists  of  summary  informa¬ 
tion  and  "is  a  scaled-down  version  of  Format  1  of  the  C/SCSC"  (13:3). 

The  two  major  parts  of  Format  1  of  the  C/SSR  are  Contract  Data  and 
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Performance  Data.  The  contract  data  determines  overall  contract  value  so 


that  a  comparison  can  be  made  to  the  baseline.  The  performance  data 
reports  the  status  of  the  contract  for  specific  WBS  items  on  a 
cumulative-to-date  basis.  The  C/SSR  also  includes  the  contractor's 
latest  revised  estimate  (LRE)  of  cost  at  completion.  And  like  the  CPR, 
the  C/SSR  is  not  usually  applied  to  firm-fixed  price  contracts. 

There  are  essentially  four  important  differences  between  the  CPR  and 
the  C/SSR.  First,  the  C/SSR  does  not  require  performance  reporting  by 
functional  area  whereas  it  is  required  in  the  CPR.  Second,  as  stated  in 
the  above  paragraph,  WBS  performance  is  reported  on  a  cumulative-to-date 
basis  for  the  C/SSR.  Therefore,  current-period  or  incremental  informa¬ 
tion  for  ACWP,  BCWP,  BCWS,  etc.  are  not  provided  by  the  C/SSR  whereas  the 
CPR  does  include  this  information.  Third,  baseline  reporting  and  projec¬ 
tions  for  manpower  loading  are  not  required  for  the  C/SSR  but  are 
required  for  the  CPR.  Fourth,  the  definition  of  BCWS  and  BCWP  are 
different  for  the  C/SSR  and  the  CPR.  Whereas  the  CPR  requires  that  BCWS 
and  BCWP  are  amounts  determined  by  sunning  work  package  budgets ,  the 
C/SSR  determines  BCWS  and  BCWP  amounts  by  methods  other  than  work 
packages.  The  method  used  for  the  C/SSR  is  negotiated  between  the  DOD 
project  manager  and  the  contractor.  Therefore,  the  C/SSR  allows  greater 
flexibility  than  the  CPR  in  selecting  an  internal  performance  measurement 
system  (13:3). 

How  to  Evaluate  the  CPR.  In  evaluating  how  a  contractor  is  perform¬ 
ing  on  its  contract,  the  two  major  areas  that  are  considered  are  cost  and 
schedule.  Contractor  data  used  in  evaluating  cost  and  schedule  perfor¬ 
mance  include  the  Budgeted  Cost  of  Work  Scheduled  (BCWS),  Budgeted  Cost 
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of  Work  Performed  (BCWP),  Actual  Cost  of  Work  Performed  (ACWP),  Cost 
Variance  (CV),  Schedule  Variance  (SV)  and  Estimate  at  Completion  (EAC) . 

As  mentioned  before,  cost  and  schedule  performance  are  recorded  on  the 
CPR  and  the  C/SSR.  The  contractor  is  required  to  establish  detailed  work 
packages  for  all  known  work  accomplished  in  the  previous  six  months.  A 
monthly  budget  is  then  developed  for  each  month  of  the  contract  based  on 
the  start  and  stop  dates  of  each  work  package  and  on  the  budgets  assigned 
to  them.  At  the  end  of  each  reporting  period,  the  contractor  reports  the 
dollar  amount  of  work  budgeted  for  each  month  for  the  CPR.  The  con¬ 
tractor  also  reports  the  cumulative  dollar  amount  budgeted  to  date  for 
the  CPR  and  the  C/SSR.  The  dollar  amount  of  work  budgeted  to  be 
performed  is  the  Budgeted  Cost  of  Work  Scheduled  to  be  accomplished  or 
BCWS f  BCWS  can  also  be  stated  as:  Of  the  work  I  scheduled  to  have  done, 
how  much  did  I  budget  for  it  to  cost? 

The  contractor  is  also  required  to  provide  the  budgeted  dollar  value 
of  work  completed  during  the  current  monthly  period.  This  is  referred  to 
as  the  Budgeted  Cost  of  Work  Performed  or  BCWP.  BCWP  can  also  be  stated 
as:  Of  the  work  I  actually  performed,  how  much  did  I  budget  for  it  to 
cost  (not  how  much  did  it  actually  cost)?  The  Schedule  Variance  (SV)  is 
the  difference  between  the  BCWP  and  BCWS.  Therefore,  the  SV  is  the 
amount  of  work,  in  dollars,  that  the  contractor  is  either  ahead  or  behind 
schedule . 

The  Actual  Cost  of  Work  Performed,  or  ACWP,  is  also  provided  by  the 
contractor  for  each  reporting  period.  ACWP  can  also  be  stated  as:  Of 
the  work  I  actually  performed,  how  much  did  it  actually  cost?  The  Cost 
Variance  (CV)  is  the  difference  between  the  BCWP  and  ACWP. 
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The  contractor  must  also  provide  an  estimate  for  costs  at  the 
completion  of  the  contract.  This  is  referred  to  as  Estimate  at  Comple¬ 
tion  or  EAC.  There  are  a  variety  of  ways  to  calculate  the  EAC  as  will  be 
seen  in  the  next  section  titled  "Statistical  Measures  of  Contract  Status. 

As  stated  before,  all  costs  (budgeted,  actual  and  estimated)  are 
required  to  be  sunmarized  both  by  Work  Breakdown  Structure  or  WBS  (Format 
1  of  the  CPR)  and  by  the  contractor's  functional  or  organizational  struc¬ 
ture  (Format  2  of  the  CPR).  The  WBS  is  defined  in  MIL-STD-881A  and  "is  a 
product  oriented  tree-like  structure  that  relates  the  hardware,  software, 
services  and  other  work  tasks  required  during  development  and  production" 
(13:2).  The  contractor  is  required  to  summarize  his  performance  down  to 
the  third  level  of  the  WBS  and  must  be  able  to  identify  variances  at 
least  at  the  fourth  level,  or  one  level  below  that  reported  to  the 
government.  Thus,  the  contractor  must  not  only  be  able  to  identify  the 
work  package(s)  associated  with  a  particular  variance  but  must  also  know 
the  reason  or  cause  of  that  variance.  This  can  be  easily  accomplished  by 
working  downwards  through  the  data  in  the  WBS  format  until  the  precise 
area  for  the  cause  of  the  variance  can  be  determined.  Also,  by  reporting 
at  a  summary  level  on  the  CPR  and  C/SSR,  small  variances  will  cancel  out 
whereas  large  variances  will  appear  on  the  reports.  This  enables  the 
analyst  to  know  where  to  begin  searching  for  the  causes  of  the  variances 
(11:162). 

Various  formulas  have  been  derived  to  evaluate  contractor  cost  and 
schedule  performance.  It  is  not  only  important  to  know  what  information 
is  contained  in  the  formulas,  but  also  to  be  able  to  interpret  the 
information  that  the  formulas  provide: 
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Statistical  Measures  of  Contract  Status 

1.  Cost  Performance  Index  [for  Efficiency]  (CPIg):  The  CPIg 
is  an  indicator  of  the  cost  efficiency  at  which  work  is  being 
accomplished.  [It  indicates  with  what  efficiency  the  contractor  is 
staying  within  his  budget.  (11:322)]  It  is  defined  as  the  ratio  of 
BCWP  to  ACWP  expressed  as  a  percent.  (13:4) 

CPIg*  =  BCWP  x  100 
ACWP 

Calculation  of  CPIg  can  be  made  for  both  incremental,  i.e.  monthly, 
or  cumulative-to-date  data.  The  CPIg  is  interpreted  to  mean  that 
for  every  budget  dollar  spent  a  CPI  percent  return  value  is 
received.  Thus,  a  CPI  of  90S  indicates  that  for  every  dollar  spent 
only  90  cents  in  value  is  being  received.  An  efficiency  or  CPI  of 
100  means  that  the  contract  is  on  target.  A  CPI  of  less  than  100 
indicates  poor  efficiency  and  cost  overrun.  A  CPI  of  greater  than 
100  indicates  good  efficiency  and  cost  underrun.  (13:4) 

[Cost  Performance  Index  Profile:  CPIp  =  ACWP 

BCWP 

Indicates  the  actual  cost  of  each  one  dollar's  worth  of  work 
accomplished.  (11:322)] 

2.  Schedule  Performance  Index  [for  Efficiency]  (SPIg):  The 
SPIg  is  an  indicator  of  the  schedule  efficiency  at  which  work  is 
being  accomplished,  [it  indicates  with  what  efficiency  the 
contractor  is  st.  ring  within  his  schedule  plan  for  getting  the  work 
done.  (11:322)]  It  is  defined  as  the  ratio  of  BCWP  to  BCWS 
expressed  as  a  percent.  (13:4) 

SPIgZ  =  BCWP  x  100 
BCWS 

Calculation  of  SPIg  can  be  made  for  both  incremental,  i.e.  monthly, 
or  cumulative**to~date  data.  The  SPI  is  interpreted  to  mean  that 
work  is  proceeding  at  a  rate  given  by  the  SPI.  Thus,  an  SPI  of 
100X  indicates  that  work  is  on  schedule.  An  SPI  greater  than  100Z 
indicates  an  ahead-of-schedule  condition.  (13:4) 

3.  Cost  Variance  (CV):  [The  difference  between  budgeted  costs 
and  actual  costs  (for  all  work  actually  performed)  is  a  variance  in 
costs.  (11:30)]  The  cost  variance  is  expressed  as  both  a  dollar 
value  and  a  percent  as  follows:  (13:4) 

CV($)  =  BCWP  -  ACWP 

CV(%)  =  CV(S)  x  100 
BCWP 

The  cost  variance  expressed  as  a  dollar  value  does  not  always 
indicate  the  true  magnitude  of  the  problem.  The  significance  of 
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this  statistic  can  often  be  more  clearly  displayed  by  use  of 
comparison  to  some  base.  This  base  is  the  cost  of  work  accomplished 
or  BCWP,  and  is  used  in  the  calculation  of  the  cost  variance 
percent.  The  cost  variance  can  be  calculated  on  either  an 
incremental  or  cumulative-to-date  basis.  The  cost  variance  percent 
can  also  be  expressed  as:  (13:4) 

CV[%]  =100-100 

CPI[E] 

4.  Schedule  Variance  (SV):  [The  difference  between  work 
scheduled  and  work  performed  (in  terms  of  budgeted  cost)  is  a 
variance  in  schedule.  (11:29)]  The  schedule  variance  is  expressed 
as  a  dollar  value,  a  percent,  and  also  a  months  ahead/behind 
statistic  as  follows:  (13:4-5) 

SV($)  =  BCWP  -  BCWS 

SV(t)  =  SV(S)  x  100 
BCWS 

SV(months)  =  SV(S) 

BCWS 

The  schedule  variance  expressed  as  a  dollar  value  does  not 
•always  indicate  the  true  magnitude  of  the  problem.  The  significance 
of  this  statistic  can  often  be  more  clearly  displayed  by  comparison 
to  some  base.  This  base  is  the  cost  of  work  scheduled  or  BCWS,  and 
is  used  in  the  calculation  of  the  schedule  variance  percent.  The 
cost  variance  can  be  calculated  on  either  an  incremental  or 
cumulative-to-date  basis.  The  schedule  variance  percent  can  also  be 
expressed  as:  (13:5) 

SV(%)  =  SPI[(100)]  -  100 

Negative  values  for  the  schedule  variance  statistic  indicate  an 
unfavorable,  that  is,  a  behind-schedule  position. 

5.  Percent  Complete:  This  is  defined  as  the  relationship 
between  work  accomplished  to  date  (BCWP)  and  budgeted  total  amount 
of  work  planned.  The  usual  base  in  determining  percent  complete  is 
the  Budget  at  Completion  (BAC)  as  shown  on  the  CPR.  (13:5) 

Percent  Complete  =  BCWP  x  100 

BAC 

6.  Percent  Spent:  Defined  as  the  relationship  between  dollar 
amounts  actually  spent  to  date  (ACWP)  to  the  budgeted  total  amount 
of  work  planned.  The  base  in  determining  percent  spent  is  the  BAC. 
The  Latest  Revised  Estimate  may  also  be  used.  (13:5) 
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Percent  Spent  =  ACWP  xlOO 
BAC 

Comparison  of  Percent  Complete  to  Percent  Spent  is  one  method 
of  determining  the  financial  state  of  the  contract.  When  percent 
spent  exceeds  percent  complete,  it  indicates  that  actual  cost  of 
work  exceeds  the  budgeted  cost  of  the  work.  This  comparison  is 
similar  in  nature  to  the  reasoning  behind  the  use  of  the  CPI.  (13:5) 

To  Complete  Performance  Index  (TCPI):  This  index  is  defined  as 
the  CPI  that  must  be  attained  to  have  the  contract  meet  the  planned 
budget.  (13:5) 

TCPIp  =  work  remaining  =  (BAC  -  BCWPc<mt)  x  100 
money  remaining  (BAC  -  ACWPrim>) 

Whereas  the  CPIg  indicates  past  fiscal  efficiency,  the  TCPI 
indicates  the  budget/cost  efficiency  at  which  the  contractor  must 
perform  in  the  future  to  complete  the  contract  within  the  targeted 
BAC.  The  CBB  or  EAC  may  be  substituted  for  the  BAC 
in  the  denominator  to  indicate  the  future  efficiency  needed  to 
complete  the  contract  within  their  respective  target  goals.  (11:322) 

In  addition  to  having  only  one  formula  used  in  calculating  a  certain 

statistic  (as  in  the  above  examples),  many  different  formulas  may  also 

exist  for  calculating  a  single  performance  parameter.  The  Estimate  at 

Completion  (EAC)  is  an  example  of  this: 

Latest  Revised  Estimate  of  Cost  at  Completion  (LRE/EAC):  All 
formulae  for  calculating  LRE/EAC  must  take  into  account  all  actual 
costs  already  expended  and  the  amount  of  work  remaining  to  be 
accomplished.  EAC  formulae  may  only  differ  in  their  projections  of 
the  future  work  environment  (i.e.  with  what  efficiency  future  work 
will  be  accomplished).  (11:322) 

a.  EAC  =  ACWP  ♦  CPI?(BAC  -  BCWPcup)  =  CPIp(BAC) 

[This  is  true  only  for  cumulative  data  in  the  CPI.] 

b.  EAC  =  ACWP  ♦  (BAC  -  BCWP  )  =  BAC 

(  CPIE  )  CPIg 

[This  is  true  only  for  cumulative  data  in  the  CPI.] 

c.  EAC  ♦  ACWP^ -  =  ACWP  _ 

BCWPcu(#/BAC  Z  complete 

NOTE:  All  the  above  formulae  will  give  the  same  EAC;  each  is  equal 
to  the  other.  All  are  projecting  the  EAC  based  upon  a  past 
efficiency  factor.  [This  is  true  only  for  cumulative  data 
in  the  CPI . ] 
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d.  EAC  =  ACWP 


+  ( BAC  -  BCWP _  _  ) 

, - cum' 

(  spie  ) 

NOTE:  This  EAC  is  based  upon  the  past  work  (schedule)  efficiency. 


e . 


EAC  =  ACWP  + 
cum 


LB_AC  -,bcwp _ 

(  .  5(SPIg)  +  75(CPIg) 


) 

) 


NOTE:  This  EAC  is  based  upon  an  equal  weighting  of  the  past 

schedule  and  budget  efficiency  factors.  Different  weighting 
may  be  substituted  Lor  the  50/50  spread  shown  above.  This 
weighting  of  SPIg  and  CPIg  is  an  SCIg  (Schedule/Cost  Index  of 
Efficiency).  (11:322) 


If  the  evaluator  determines  that  significant  variances  exist  in  the 
CPR  data,  an  examination  of  the  contractor  problem  analysis  section  may 
reveal  the  cause  of  the  problems: 

When  significant  variances  exist,  the  CPR  or  C/SSR  should 
contain  a  problem  analysis  page.  This  page  should  be  reviewed  to 
determine  if  the  contractor  has  provided  the  following:  specific 
causes  for  the  significant  variances,  the  action  that  is  needed  to 
be  taken  to  correct  the  situation,  and  the  period  of  time  needed  to 
correct  the  problem.  This  narrative  should  be  clear  and  complete; 
if  it  is  not,  then  the  contractor  should  be  required  to  submit  a 
more  coherent  analysis.  The  information  in  this  narrative  is  the 
type  of  information  that  should  be  blended  with  the  statistical 
analysis  to  provide  a  more  comprehensive  picture  of  the  trends  and 
changing  nature  of  the  contract.  (13:6) 

CPR  Software  Analysis  Packages.  As  stated  previously,  the  data  on 
Cost  Performance  Reports  (CPRs)  are  used  to  analyze  the  contractor's 
performance  in  terms  of  dollars  (cost)  and  time  (schedule)  required  to 
complete  the  work.  In  order  to  analyze  contractor  performance,  many 
software  programs  have  been  developed  to  evaluate  the  data  on  CPRs .  A 
list  and  description  of  the  CPR  software  packages  that  will  be  analyzed 
is  discussed  in  Appendix  A.  The  software  packages  that  will  be  analyzed 
include  the  following: 
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Space  Systems  Division  (SSD)  Performance  Analyzer  (PA) 

Air  Force  Cost  Center  CPR-EZ 

Air  Force  Cost  Center  Contractor  Cost  Analysis  System  (CCAS) 

Defense  Systems  Management  College  (DSMC)  Contract  Appraisal 

System  (CAPPS) 

Criteria  for  Selecting  Software  Packages.  In  evaluating  and  compar¬ 
ing  software  packages,  the  evaluator  must  have  a  set  of  criteria  with 
which  to  compare  the  software  packages.  The  objective  of  this  study  is 
to  first  determine  the  criteria  needed  to  conduct  CPR  software  analysis. 
Then  the  different  types  of  software  can  be  evaluated  against  these 
criteria  for  performing  CPR  analysis. 

In  determining  how  to  1 1»«- 1  these  criteria  for  conducting  CPR  soft¬ 
ware  analysis,  criteria  used  previously  in  selecting  software  packages 
for  functions  other  than  CPR  analysis  will  be  reviewed.  Various  software 
packages  have  been  selected  based  upon  certain  criteria  that  are  appli¬ 
cable  to  all  types  of  software  and  other  criteria  that  are  applicable 
only  to  the  type  of  software  being  acquired.  Therefore,  criteria  used  in 
selecting  the  following  software  packages  may  or  may  not  be  the  same  as 
the  criteria  that  will  be  used  in  selecting  CPR  software  analysis 
packages.  The  following  studies  will  be  evaluated  with  regard  to  select¬ 
ing  criteria  for  evaluating  CPR  software  packages:  a  student  thesis 
regarding  criteria  used  in  selecting  project  management  software;  a 
review  of  AFSCP  800-14,  Air  Force  Systems  Command  Software  Quality 
Indicators;  criteria  used  in  selecting  educational  operations  research 
software;  and  a  PC  Magazine  test  comparing  55  word  processors  against 
nine  criteria. 
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Proiect  Management  Study.  Capt.  Robert  J.  Hartnett,  Jr.,  a  GSM 
(Graduate  of  Systems  Management)  student  of  AFIT  class  of  1989S,  wrote  a 
thesis  titled  Proiect  Management  Software:  Proper  Selection  for  Use  Within 
Air  Force  Systems  Command.  In  his  thesis,  Capt.  Hartnett  used  four 
criteria  for  his  "minor  project"  and  twelve  criteria  to  evaluate  his 
"major  project". 

The  four  criteria  used  in  Capt.  Hartnett's  "minor  project"  were 
installation,  tutorial,  data  entry  and  reporting.  Installation  refers  to 
whether  the  system  can  be  installed  quickly  and  easily  without  thorough 
knowledge  of  how  the  operating  system  works.  A  helpful  tutorial  must 
exist  that  can  be  used  easily  by  both  beginning  and  experienced 
operators.  Data  must  be  able  to  be  manipulated  in  many  different  ways 
when  the  software  system  will  be  used  by  a  variety  of  people  in  different 
jobs.  The  data  must  be  able  to  be  entered  quickly  and  easily  and  must 
consider  dependencies  between  various  tasks,  resource  information  and 
cost  data.  It  is  essential  to  have  a  reporting  capability  in  which 
reports  can  be  viewed  on  the  computer  screen  and  inmediately  sent  to  a 
printer  for  a  hardcopy  printout.  The  four  criteria  listed  above  must  be 
easily  learned  and  performed  so  that  the  software  packages  allow  the  user 
to  quickly  become  productive.  This  is  just  as  important  as  the  system 
being  capable  of  performing  those  functions. 

Capt.  Hartnett's  "major  project"  involved  using  software  packages  to 
plan,  track  and  report  on  a  government  defense  contract.  As  stated 
above,  the  "major  project"  included  twelve  criteria.  Some  of  these  cri¬ 
teria  apply  to  CPR  software  analysis  packages  and  some  do  not.  The 
twelve  criteria  are  as  follows:  project  modeling  capability,  scheduling 
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capability,  resource  management,  cost  management,  tracking  and  control¬ 
ling  existing  projects,  reporting  capability,  advanced  project  management 
functionality,  documentation,  ease  of  learning,  ease  of  use,  vendor  and 
cost . 

The  first  criterion,  project  modeling  capability,  is  divided  into 
two  parts — capacity  of  system  and  extensiveness  of  options  or  features. 
Project  modeling  capability  must  allow  for  easy  entry  of  incomplete 
project  data  and  insertion  of  additional  information  and  updating  proce¬ 
dures.  Capacity  of  the  system  refers  to  the  number  of  activities  allowed 
per  project  and  the  number  of  dependencies  allowed  per  activity.  Exten¬ 
siveness  of  options  or  features  refers  to  whether  the  managers  are  forced 
to  model  their  projects  in  a  certain  way  or  if  they  can  choose  their  own 
style  of  modeling. 

The  second  criterion,  scheduling  capability,  includes  the  ability  to 
present,  produce  and  manipulate  scheduling  information  so  that  resources 
and  activities  can  be  examined  and  modified  easily.  Hartnett  used  the 
following  three  areas  to  describe  this  criteria:  schedule  type  (Gantt, 
PERT);  scheduling  time  base  (can  activities  be  viewed  by  hours,  days, 
weeks,  months  and  years?);  and  accessibility  of  information  (can  the 
project’s  logic  with  regard  to  precedence  relationships  and  the  activity 
details  be  easily  examined  and  modified?). 

Resource  management,  the  third  criterion,  includes  planning,  level¬ 
ing  and  tracking.  Resource  management  refers  to  whether  or  not  the 
system  can  efficiently  allocate  scarce  resources  to  different  projects 
which  require  the  use  of  those  resources.  It  includes  being  able  to 
handle  individual  resource  calendars  (planning)  as  well  as  allocating 
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resources  to  conflicting  activities  (leveling)  while  ensuring  all 
resources  are  being  used  as  efficiently  as  possible  (tracking). 

Cost  management,  the  fourth  criterion,  is  divided  into  planning, 
programming ,  budgeting  and  tracking.  Cost  management  compares  the 
required  funding  to  the  approved  funding  for  a  project.  Therefore,  it  is 
essential  that  a  project  management  software  package  includes  information 
to  plan,  allocate  and  track  funds  to  different  projects  and  activities. 

The  fifth  criterion,  tracking  and  controlling  existing  projects, 
involves  comparing  revised  or  actual  data  with  the  original  planned 
information  and  activity  using  this  information  to  keep  the  project  on 
schedule.  This  criterion  consists  of  data  entry  and  analysis. 

Reporting  capability  involves  the  ability  to  provide  reports  that 
are  easy  to  read  and  understand  with  varying  levels  of  detail.  The 
reports  must  offer  summary  detail,  graphics  and  quality  that  is  suffi¬ 
cient  for  presentation  purposes.  This  criterion  includes  customized 
reports  and  quality  of  output.  Customized  reports  refer  to  whether  the 
report  formats  can  be  changed,  if  reporting  can  be  performed  only  on 
items  of  interest  and  if  sorting  can  be  accomplished  in  a  variety  of 
ways.  Quality  of  output  refers  to  whether  reports  can  be  viewed  on  the 
computer  screen,  whether  the  printed  output  is  of  presentation  quality 
and  if  the  software  package  supports  plotter  graphics. 

Advanced  project  management  functionality  is  the  seventh  criterion 
and  includes  baselining,  analyses  and  interfacing  with  other  programs  and 
computers.  This  criterion  deals  with  being  able  to  compare  alternatives 
by  baselining.  It  also  includes  being  able  to  obtain  an  optimum  solution 
by  conducting  "what  if"  analyses  and  risk  assessments.  It  is  also  impor- 
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tant  for  the  system  to  be  able  to  interface  with  other  programs  and 
computers  by  importing  and  exporting  different  file  formats  and  being 
able  to  be  used  over  a  Local  Area  Network  (LAN). 

Documentation  must  not  only  be  understandable  but  also  usable.  In 
order  to  be  able  to  use  the  software  to  its  utmost  potential,  the  user 
must  be  able  to  obtain  answers  to  questions  quickly  and  easily.  This  can 
be  accomplished  if  the  documentation  is  clear  and  well-written  so  that  it 
will  be  understood  by  all  users,  especially  the  non-technical  user.  All 
information  should  be  thoroughly  indexed  so  that  the  user  can  find  infor¬ 
mation  quickly  and  easily. 

Ease  of  learning,  the  ninth  criterion,  includes  a  hands-on  tutorial 
and  productivity.  Because  people  quickly  become  frustrated  with 
computers  when  they  are  hard  to  learn  how  to  use,  it  is  important  that 
the  learning  process  is  as  easy  as  possible  for  the  user.  A  hands-on 
tutorial  enables  the  user  to  learn  by  performing  the  task.  This  normally 
results  in  the  user  learning  how  to  perform  the  task  more  quickly.  The 
user  will  also  become  less  frustrated  if  he  or  she  can  become  productive 
quickly  on  the  software  instead  of  having  to  learn  how  to  use  the  entire 
software  package  before  it  can  be  used  correctly. 

The  tenth  criterion  is  ease  of  use  and  includes  screen  color  mix, 
program  hierarchy,  command  presentations  and  a  "help"  function.  If  a 
software  package  is  easy  to  use  and  understand,  the  worker  will  become 
more  productive  and  will  enjoy  using  the  software  package.  For  example, 
a  user  is  more  apt  to  use  a  software  package  that  includes  a  screen  color 
mix  that  is  aesthetically  pleasing  and  can  be  easily  changed  than  one 
that  is  offensive  to  the  eyesight.  Program  hierarchy  refers  to  whether 
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the  command  structure  is  logical  in  that  there  is  an  input-processing- 
output  format.  It  also  refers  to  whether  an  operation  requires  a  large 
number  of  commands  in  order  to  be  performed.  Command  presentations  refer 
to  whether  a  menu  system  or  function  keys  are  used  as  well  as  whether  or 
not  all  possible  options  are  displayed  for  review  before  a  certain 
command  is  performed.  The  "help"  function  includes  whether  help  is 
available  for  context-sensitive  information  and  whether  error  messages 
are  useful  in  that  they  direct  the  user  to  where  he  or  she  can  receive 
further  assistance. 

When  acquiring  a  software  package,  the  user  will  want  to  select  a 
vendor  that  is  technically  competent  and  easy  to  work  with.  Therefore, 
the  user  will  want  to  consider  experience,  technical  assistance,  imple¬ 
mentation,  training  and  learning  from  user  feedback.  With  regard  to 
experience,  the  user  will  want  to  look  at  the  amount  of  time  the  vendor 
has  been  in  business  and  whether  the  company  is  well-managed  and  finan¬ 
cially  sound.  The  user  will  also  want  to  consider  how  quickly  the  vendor 
is  able  to  answer  the  user's  questions  and  whether  the  vendor  can  answer 
specific  application  questions  (i.e.,  CPR-related  questions)  as  well  as 
questions  about  the  features  of  the  software  package.  A  vendor  that  is 
able  to  employ  high-quality  consultants  to  implement  and  support  the 
software  package  is  preferable  to  one  who  cannot  do  this.  Likewise,  a 
vendor  which  provides  different  levels  of  training  programs  for  novice, 
intermediate  and  expert  users  is  also  preferable  to  one  who  does  not  do 
this.  Another  important  aspect  of  choosing  a  vendor  is  whether  the 
vendor  learns  from  user  feedback  and  implements  user  suggestions  in  new 
releases  of  the  software  package.  The  user  will  also  want  to  consider 


23 


whether  the  updated  versions  were  adequate  for  the  user's  needs  and  if 
the  vendor  made  the  new  releases  available  when  they  were  needed. 

The  twelfth  and  final  criterion  is  cost.  There  may  be  a  wide  range 
of  costs  depending  on  what  software  packages  are  available  and  what  fea¬ 
tures  are  essential  to  get  the  job  accomplished.  Capt.  Hartnett  assigned 
various  grades  to  the  different  price  ranges  (i.e.,  excellent  for  $0  to 
$500,  very  good  for  $501  to  $1000,  etc.)  (8:50-65). 

APSC  Software  Quality  Indicators.  AFSCP  800-14,  Air  Force 

Systems  Coomand  Software  Quality  Indicators,  outlines  the  following  list 

(with  associated  definitions)  of  eleven  software  quality  factors: 

Correctness .  The  degree  to  which  the  software  satisfies  its 
specified  requirements. 

Efficiency.  The  degree  to  which  the  software  performs  its  intended 
functions  with  minimum  consumption  of  computer  time  and  storage 
resources . 

Flexibility.  The  effort  required  to  enhance  the  software  or  to 
modify  it  to  meet  n$w  requirements.  Flexibility  and  maintainability 
make  software  supportable. 

Integrity.  The  degree  to  which  the  software  controls  unauthorized 
access  to  or  modification  of  system  software  and  data. 
Interoperability.  The  degree  to  which  the  software  is  able  to 
interface  with  other  systems. 

Maintainability.  The  effort  required  to  locate  and  correct  any 
error  in  the  software.  Maintainability  and  flexibility  make 
software  supportable. 

Portability.  The  effort  required  to  transfer  the  software  from  one 
hardware  or  software  environment  to  another. 

Reliability.  The  degree  to  which  the  software  consistently  performs 
its  intended  functions. 

Reusability.  The  degree  to  which  the  software  can  be  used  in 
multiple  applications. 

Testability.  The  effort  required  to  ensure  that  the  software 
performs  its  intended  functions. 

Usability.  The  effort  required  to  learn  the  human  interface  with 
the  software,  to  prepare  input,  and  to  interpret  output  of  the 
software.  (4:36) 

Operations  Research  Study.  The  criteria  used  in  a  study  for 
selecting  educational  operations  research  software  included  "accuracy, 
speed,  ease  of  use,  ease  of  learning  and  caliber  of  documentation" 
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(15:65).  The  following  criteria  for  each  category  were  determined  from 

the  article: 

1 .  accuracy 

a.  number  of  decimal  places  in  result 

b.  round-off  errors 

2 .  speed 

a.  amount  of  time  it  takes  the  software  to  accept,  process  and 
execute  the  data 

b.  whether  the  software  keeps  up  with  the  data  being  entered  or 
causes  errors  because  of  not  accepting  the  data  readily  as  it  is 
entered 

3.  ease  of  use  ("user  friendly") 

a.  ability  to  move  around  the  spreadsheet  easily  (can  only  input 
information  one  cell  at  a  time  versus  being  able  to  jump  across 
rows  or  columns;  data  entered  in  response  to  prompting  versus 
flexibility  of  inputting  various  information) 

b.  if  the  software  uses  prompts,  are  there  words  with  the  prompts 
so  that  the  user  knows  what  information  to  input? 

c.  can  information  be  edited  at  any  time  (correcting  errors  is 
easy)  or  can  corrections  be  made  only  after  all  data  is  entered 
using  the  "edit"  function? 

d.  if  there  is  an  error  message,  does  the  software  show  what  type 
of  error  has  been  made? 

e.  flexibility  of  software  (i.e.  avoid  having  user  respond  "yes"  or 
"no"  as  to  whether  the  number  he/she  just  entered  is  correct  or 
not;  software  accepts  inputs  that  are  either  in  lower  case  or 
capital  letters  and  does  not  require  that  inputs  be  made  using 
the  "CAPS"  mode  at  all  times;  if  software  uses  prompts,  it  does 
not  require  "yes"  or  "no"  responses  to  be  spelled  out 
completely) 

f.  capability  to  add  and/or  delete  data 

4.  ease  of  learning 

a.  easy-to-follow  instructions 


25 


b.  assumes  little  previous  computer  knowledge,  software  program 
knowledge  and  subject  area  (CPR)  knowledge 

c.  includes  a  "helpful"  tutorial 
5.  caliber  of  documentation 

a.  assumes  little  previous  computer  knowledge,  software  program 
knowledge  and  subject  area  (CPR)  knowledge 

b.  includes  sections  on  "what  to  do  if  it  does  not  work" 

c.  includes  explanations  of  error  messages 

d.  includes  adequate  explanations  on  how  to  use/enter  data 

e.  includes  examples  for  all  features  and  uses;  user  is  carefully 
led  through  examples 

f.  all  features  are  listed  and  explained  in  documentation 

g.  instructions  are  clearly  written  and  easy  to  read  and  follow 

h.  includes  index 

i.  includes  all  of  the  features  that  the  documentation ‘lists 

Word  Processor  Study.  In  February  1988,  PC  Magazine  conducted 
a  test  in  which  the  following  nine  criteria  were  used  in  comparing  55 
word  processors:  the  import  file  size  test,  document  manipulation  test, 
global  search  and  replace  test,  cursor  movement  test,  search  for  middle 
test,  block  manipulation  test,  the  merge  test,  the  text  delete  test  and 
the  print  whole  document  test. 

The  import  file  size  test  measured  the  size  of  the  final  imported 
file  after  the  software  program  brought  a  50K  document  into  its  own  envi¬ 
ronment.  The  imported  file  size  varies  because  programs  manipulate  files 
differently  in  the  way  formatting  codes  are  inserted  into  the  document. 
Bight  of  the  fifty-five  word  processing  programs  were  unable  to  import  a 
file  the  size  of  50K  and  one  program  was  unable  to  read  the  ASCII  file 
because  it  used  a  non-DOS  format  (12:278). 
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The  document  manipulation  test  was  based  on  the  time  it  took  each 
word  processor  to  load  and  save  a  file.  This  test  was  divided  into  two 
parts:  document  load,  and  save  and  resume. 

The  global  search  and  replace  test  measured  the  length  of  time  re¬ 
quired  to  replace  the  work  "his"  with  the  word  "theirs".  The  test  that 
was  conducted  required  the  word  "his"  to  be  replaced  115  times. 

The  cursor  movement  test  determined  the  length  of  time  required  to 
move  the  cursor  through  the  document.  This  test  was  conducted  in  two 
parts:  a  cursor  top-to-middle  test  and  a  cursor  middle-to-bottom  test. 

The  search  for  middle  test  measured  the  length  of  time  required  to 
locate  a  sentence  in  the  middle  of  the  document .  The  test  that  was  con¬ 
ducted  allowed  for  searching  on  a  single  keyword  or  string  of  words 
because  some  word  processors  are  unable  to  search  for  an  entire  sentence. 

The  block  manipulation  test  was  based  on  a  block-and-move  test  and  a 
block-and-copy  test.  The  block-and-move  test  measured  the  time  it  took 
to  highlight  the  first  three  pages  of  the  document  and  then  to  move  it  to 
the  middle  of  the  document.  The  block-and-copy  test  measured  the  time  it 
took  to  copy  the  same  three  pages  to  an  external  file.  Because  both 
tests  required  the  pages  to  be  renumbered,  the  performance  of  programs 
with  smaller  RAM  buffers  was  slower  than  programs  with  larger  buffers. 

The  merge  test  measured  the  amount  of  time  it  took  the  program  to 
import  an  external  file  to  where  it  was  positioned  immediately  after  the 
middle  of  the  document.  As  in  the  block  manipulation  test,  the  perfor¬ 
mance  of  the  programs  was  affected  by  reformatting  the  pages. 

The  text  delete  test  measured  the  length  of  time  required  to  high¬ 
light  and  delete  the  file  that  was  imported  by  the  merge  test.  Again, 
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page  reformatting  affected  the  performance  of  programs  with  smaller 
buffers . 

The  print  whole  document  test  measured  how  long  it  took  the  word 
processor  to  print  the  50K  document  to  an  external  file  (12:279). 

Summary  of  the  Literature 

The  primary  report  that  the  government  requires  contractors  to 
submit  for  the  purpose  of  monitoring  cost  and  schedule  performance  is  the 
Cost  Performance  Report  (CPR).  In  order  to  analyze  contractor  perfor¬ 
mance  ,  many  software  programs  have  been  developed  to  evaluate  the  data  on 
CPRs.  In  determining  which  of  those  software  packages  is  the  ideal  evalu¬ 
ator  of  CPR  data,  certain  criteria  need  to  be  established  for  evaluating 
those  software  packages.  These  criteria  must  be  validated  not  only  by 
the  users  of  those  packages,  but  also  by  the  evaluators  of  the  work  that 
the  users  perform.  Once  the  software  packages  are  evaluated  against 
these  criteria,  "the  ideal"  software  package  for  evaluating  CPRs  can  be 
determined.  If  "an  ideal"  software  program  for  evaluating  CPRs  does  not 
currently  exist,  a  software  program  can  then  be  developed  which  includes 
the  features  or  criteria  that  have  been  chosen  for  accomplishing  this 
objective.  The  developers  of  the  software,  along  with  the  users  and  the 
evaluators,  will  need  to  determine  what  features  should  be  included  in 
the  software  in  order  to  meet  the  chosen  criteria.  They  also  need  to 
discuss  how  those  features  can  be  constructed  so  that  the  user  is  able  to 
understand  how  to  use  the  software  in  performing  his  or  her  work. 

Criteria  used  in  selecting  software  packages  for  use  in  project 
management,  educational  operations  research  and  word  processors  were 
reviewed  along  with  criteria  for  evaluating  software  quality  used  by  Air 
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Force  Systems  Coranand  (APSCP  800-14).  Table  1  is  a  sunmary  of  the 
criteria  used  in  selecting  those  software  packages  that  may  also  be 
applied  to  CPR  software  packages.  Criteria  that  are  application-specific, 
such  as  scheduling  capability  as  well  as  tracking  and  controlling 
existing  projects,  will  not  be  listed  because  they  apply  only  to  the 
project  management  study. 


TABLE  1 

SUMMARY  OF  SOFTWARE  CRITERIA  FOR  FOUR  DIFFERENT  STUDIES 


Project 

Management 


Criteria  Study 


Installation  x 

Tutorial  x 

Hands-on  Tutorial  x 

Data  Entry  Flexibility  x 

Reporting  x 

Customized  Reports  x 

Quality  of  output  x 

Interfacing  with  other 

programs  and  computers  x 

Understandable 

documentation  x 

Usable  documentation  x 

Easy  to  learn  (become 

productive  quickly)  x 

Easy  to  Use  -  Screen 

color  mix  x 

color  mix 

Easy  to  Use  -  Program 

hierarchy  x 

Easy  to  Use  -  Command 

presentations  x 

Easy  to  Use  -  help 

function/useful 
error  messages  x 

Vendor  experience  x 

Vendor  technical 

assistance  x 

Vendor  implementation 

and  support  x 

Vendor  -  training  prgms .  x 

Vendor  -  considers  user 

feedback  x 


AFSC  Operations  Word 

Software  Quality  Research  Processor 

Indicators  Study  Study 

x 

x 

x 

X  X 

X 

X 

X  X 

X 

X 

X 

X 
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TABLE  1 


SUMMARY  OF  SOFTWARE  CRITERIA  FOR  FOUR  DIFFERENT  STUDIES  (Continued) 


Criteria 


Project  APSC  Operations  Word 

Management  Software  Quality  Research  Processor 
Study  Indicators  Study  Study 


Cost  x 

Correctness  (satisfies 
requirements) 

Efficient 

Plexible  (effort  to 

enhance  or  modify  SW) 
Integrity  (ability  to 
control  unauthorized 
access  or  modification 
to  software) 

Maintainable  (effort  reqd. 
to  locate  and  correct 
errors  in  software) 

Portable 

Reliable 

Reusable  (multiple 
applications) 

Testable  (effort  reqd.  to 
ensure  SW  performs 
intended  functions) 
Accurateness 
Speed 


x 

X 

X 


X 


X 

X 

X 

X 


X 

X 

X  X 


In  Chapter  II,  four  different  studies  were  evaluated  with  regard  to 
identifying  criteria  used  in  selecting  various  types  of  software 
packages.  In  Chapter  III,  criteria  used  in  selecting  "the  ideal"  CPR 
analysis  software  package  will  be  identified  from  questionnaires  sent  to 
Air  Porce  Systems  Command  (APSC)  product  divisions. 
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III.  Methodology 


The  general  method  of  evaluating  CPR  analysis  software  will  involve 
using  decision  support  software  and  consists  of  two  parts.  The  first 
part  will  be  to  determine  the  criteria  necessary  to  evaluate  CPR  analysis 
software  packages.  The  second  part  will  be  to  evaluate  and  compare  CPR 
personal  computer  (PC)  software  packages  based  on  those  criteria.  Prom 
this,  "the  ideal"  software  package  or  features  of  "the  ideal"  software 
package  for  analyzing  CPR  data  will  be  determined. 

An  initial  objective  of  this  study  was  to  obtain  CPR  software 
packages  from  both  the  government  (Army,  Navy  and  Air  Force)  and 
contractors  in  order  to  compare  them  in  terms  of  similarities  and 
differences.  Because  the  following  contractors  work  with  the  government 
in  implementing  C/SCSC  and  in  developing  software  to  evaluate  CPR  data, 
they  were  contacted  to  determine  if  any  software  packages  to  evaluate  CPR 
data  were  currently  in  exist-  ace  and/or  development:  Humphreys  & 
Associates,  The  Phoenix  Group,  Decision  Planning  Corporation,  Langford  & 
Associates,  Technical  Marketing  Society  of  America  (TMSA)  and  Performance 
Management  Associates.  Information  obtained  from  these  contractors 
revealed  that  there  were  no  CPR  software  packages  currently  in 
development.  Prom  contacting  the  Army  and  Navy  Performance  Measurement 
Joint  Executive  Group  (PMjEG)  focal  points,  it  was  discovered  that  Naval 
Air  Systems  Command  is  currently  updating  ESD's  Automated  Financial 
Analysis  (AFA)  program  so  that  it  will  be  available  in  a  PC  version.  It 
is  referred  to  as  the  Automated  Financial  Analysis  (AFA)  Tabular  Report 
Generator.  The  majority  of  CPR  software  packages  that  will  be  evaluated 
are  either  currently  being  used  in  the  Air  Force  or  can  be  obtained  from 
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Defense  Systems  Management  College  (DSMC).  The  CPR  software  packages 
that  will  be  evaluated  include! 

Space  Systems  Division  (SSD)  Performance  Analyzer  (PA) 

Air  Force  Cost  Center  CPR-EZ 

Air  Force  Cost  Center  Contractor  Cost  Analysis  System  (CCAS) 

Defense  Systems  Management  College  (DSMC)  Contract  Appraisal  System 

(CAPPS) 

Phase  I!  Initial  Selection  of  Criteria  to  Evaluate  CPR  PC  Software 
As  stated  before,  in  order  to  evaluate  the  above  CPR  software 
packages,  certain  criteria  need  to  be  established  against  which  the 
software  packages  will  be  compared.  The  personnel  best  qualified  to 
establish  the  criteria  for  CPR  software  packages  are  the  users  of  that 
software.  Therefore,  a  questionnaire  was  sent  to  the  following  Air  Force 
Systems  Command  (AFSC)  product  divisions  to  request  assistance  in 
establishing  those  criteria! 

ASD/ACC  Aeronautical  Systems  Division,  Wright-Patterson  AFB,  OH 
BSD/ACI  Ballistic  Systems  Division,  Norton  APB,  CA 
ESD/ACC  Electronic  Systems  Division,  Hanscom  AFB,  MA 
HSD/ACC  Human  Systems  Division,  Brooks  AFB,  TX 
MSD/ACC  Missile  Systems  Division,  Eglin  AFB,  FL 
SSD/ACC  Space  Systems  Division,  Los  Angeles  APS,  CA 
The  information  contained  in  the  questionnaire  for  the  initial 
selection  of  the  criteria  to  be  used  in  evaluating  CPR  software  is  as 
follows! 

1.  An  evaluation  of  CPR  software  currently  being  used  by  the  various 
product  divisions  is  being  conducted  at  APIT.  The  purpose  of  this 


evaluation  is  to  determine  essential  features  that  need  to  be  included  in 


CPR  software. 

2.  In  conducting  this  evaluation,  it  is  important  that  the  following 
questions  be  answered  as  explicitly  as  possible: 

a.  What  CPR  software  are  you  currently  using? 

b.  What  features  do  you  find  useful  (please  clarify)?  (ie.  user 
friendly  -  easy  to  use  or  can  load  the  software  onto  the 
computer  with  no  problems;  documentation  -  easy  to  understand 
and  follow,  indexes  all  features;  tutorials  -  explains  all 
features  and  includes  "hands  on"  examples;  help  menus  -  includes 
additional  "help"  that  thoroughly  explains  how  to  solve  the 
problem;  easy  to  learn  -  takes  approximately  one  hour  to  learn 
how  to  use  the  software) 

c.  What  should  the  ideal  CPR  software  include? 

3.  If  you  have  any  further  suggestions  on  how  CPR  software  can  be 
improved  and/or  used,  please  attach  them  on  a  separate  sheet  of  paper. 

Phase  II;  Analysis  of  Criteria  for  Software  (Criteria  Validation  bv  Users) 

Prom  the  first  questionnaire  conducted  in  Part  I  above,  twenty-one 
criteria  were  identified.  A  second  questionnaire  was  then  administered 
requesting  the  APSC  product  divisions  to  rank  order  the  following 
criteria  established  from  the  first  questionnaire: 

(a)  help  indexes  that  tutor  on  the  analyses  instead  of  referring  to  a 
manual 

(b)  a  program  that  does  not  need  a  computer  dedicated  onto  that 
software 

(c)  a  separate  tutorial  which  includes  different  case  scenarios 

(d)  user  friendly,  menu-driven  screens 
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(e)  easy  to  use  or  can  load  the  software  onto  the  computer  with  no 
problems 

(f)  learn  as  you  use  —  get  to  work  right  away 

(g)  needs  to  be  able  to  print  on  plotter  or  printers 

(h)  needs  to  be  able  to  print  graphics  or  data 

(i)  full  CPR/C/SSR  analysis  that  uses  regression  analysis  to  predict 
trends 

(j)  variance  analysis,  with  indicators  of  major  problem  areas 

(k)  a  way  to  determine  if  data  has  been  inputted  correctly 

(l)  narrative  section  which  sums  all  data,  trends  and  variance  problems 
and  provides  suggestions  for  further  research 

(m)  documentation  -  easy  to  understand  and  follow 

(n)  ability  to  modify  formulas,  update  charts 

(o)  help  menus  -  includes  additional  "help"  that  thoroughly  explains 
how  to  solve  the  problem 

(p)  capability  to  accept  floppy  disks  from  contractors 

(q)  software  is  available  on  personal  computer 

(r)  interactive  Management  Information  System  (MIS)  with  contractor — 
contractor  CPR  data  comes  in  electronically  and  automatically  goes 
into  a  spreadsheet  to  be  analyzed 

(s)  includes  several  different  Estimate  at  Completion  (RAC) 
calculations  with  a  guide  to  determine  which  EAC  is  most 
applicable  for  a  given  program 

(t)  good  summary  report  format  that  is  easy  to  read  and  understand  in  a 
short  period  of  time 

(u)  allows  flexibility  toward  user  familiarity  with  the  subject  (i.e., 
has  an  expert  mode  to  bypass  menus) 


Phase  III:  Pinal  Selection  of  Evaluation  Criteria 

From  the  responses  by  the  AFSC  product  divisions  to  the  second 
questionnaire  administered  in  Phase  II,  the  criteria  for  evaluating  CPR 
analysis  software  packages  were  chosen.  From  studies  that  have  been 
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conducted,  it  has  been  found  that  most  people  can  only  concentrate  on 
seven  plus  or  minus  two  items  at  a  time  before  they  become  overwhelmed 
(9:81-97).  Therefore,  the  first  seven  criteria  that  the  users  have 
identified  as  being  most  important  will  be  analyzed.  Criteria  that  are 
considered  to  be  similar  in  nature  may  be  grouped  together  into  a  single 
category.  In  order  to  evaluate  and  compare  the  responses  in  a  variety  of 
ways,  three  different  comparisons  were  conducted. 

First  Comparison.  First,  the  scores  from  the  rank  order  were 
averaged  and  the  seven  criteria  were  selected  from  the  lowest  averages. 
The  results  are  as  follows: 


TABLE  2 

FIRST  COMPARISON  METHOD 


Order  of 


Criteria 

Average 

Importance 

a 

12.8 

18 

b 

13.8 

19 

c 

15.59 

20 

d 

6.41 

1 

e 

8.29 

4 

f 

10.11 

9 

e 

10.07 

8 

h 

10.22 

10 

i 

8.15 

3 

j 

7.19 

2 

k 

12.63 

16 

1 

9.93 

7 

m 

11.22 

12 

n 

9.59 

5 

o 

12.74 

17 

P 

12.26 

14 

q 

10.22 

11 

r 

12.33 

15 

s 

9.66 

6 

t 

12.07 

13 

u 

15.67 

21 
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The  seven  criteria,  in  order  of  importance,  are: 

1.  user  friendly,  menu-driven  screens 

2.  variance  analysis,  with  indicators  of  major  problem  areas 

3.  full  CPR/C/SSR  analysis  that  uses  regression  analysis  to  predict 
trends 

4.  easy  to  use  or  can  load  the  software  onto  the  computer  with  no 
problems 

5.  ability  to  modify  formulas,  update  charts 

6.  includes  several  different  Estimate  at  Completion  (EAC)  calculations 
with  a  guide  to  determine  which  EAC  is  most  applicable  for  a  given 
program 

7.  narrative  section  which  sums  all  data,  trends,  and  variance  problems 
and  provides  suggestions  for  further  research 

Second  Comparison.  Second,  the  criteria  were  selected  based  on  the 
number  of  times  each  criterion  was  rank  ordered  in  the  range  of  1  to  5. 
This  range  was  selected  because  as  stated  previously,  most  people 'can 
only  concentrate  on  seven  plus  or  minus  two  items  before  becoming 
overwhelmed.  The  seven  criteria  were  selected  from  the  highest  ratings. 
In  the  event  of  a  tie  in  the  1  to  5  range,  the  number  of  times  the 
criteria  were  ranked  in  the  range  of  1  to  6  was  considered.  The  results 
are  listed  in  Table  3  below.  The  seven  criteria  based  on  this  method  are 
as  follows  (in  order  of  importance): 

1.  user  friendly,  menu-driven  screens 

2.  variance  analysis,  with  indicators  of  major  problem  areas 

3.  full  CPR/C/SSR  analysis  that  uses  regression  analysis  to  predict 
trends 

4.  easy  to  use  or  can  load  the  software  onto  the  computer  with  no 
problems 

5.  learn  as  you  use  -  get  to  work  right  away 
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6.  interactive  Management  Information  System  (MIS)  with  contractor — 
contractor  CPR  data  comes  in  electronically  and  automatically  goes 
into  a  spreadsheet  to  be  analyzed 

7.  includes  several  different  Estimate  at  Completion  (RAC)  calculations 
with  a  guide  to  determine  which  EAC  is  most  applicable  for  a  given 
program 

The  two  other  criteria  that  tied  for  seventh  place  were: 

8.  needs  to  be  able  to  print  on  plotter  or  printers 

9.  software  is  available  on  personal  computer 


Criteria 

a 

b 

c 

*  d 
e 
f 

8 

h 

i 

j 

k 

1 

m 

n 

o 

P 

q 

r 

s 

t 

u 


TABLE  3 

SECOND  COMPARISON  METHOD 


Rank  of  1  to  5 

7 

3 

2 

15 

11 

9 

7 

6 

12 

12 

3 
6 

4 

3 

4 
4 

7 

8 
7 
3 
2 


Order  of 

Rank  of  1  to  6  Rank  of  1  to  7  Importance 


7  8  10 

5  17 

3  20 

.  1 

4 

5 

8  8 

9  11 

13  3 

15  2 

3  3  19 

7  12 

7  13 

8  16 

5  6  14 

5  5  15 

7  10  9 

6 

9  7 

3  4  18 

21 


Third  Comparison.  Finally,  the  criteria  were  selected  based  on  the 
number  of  times  each  criterion  was  rank  ordered  in  the  range  of  1  to  7. 
The  seven  criteria  were  selected  from  the  highest  ratings.  In  the  event 
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of  a  tie,  the  ranges  1  to  8  and  1  to  9  were  used,  respectively.  The 
results  are  listed  in  Table  4: 

TABLE  4 

THIRD  COMPARISON  METHOD 

Order  of 


Criteria 

Rank  of  1  to  7 

Rank  of  1  to  8 

Rank  of  1  to  9  Imoortance 

a 

8 

10 

12 

b 

7 

15 

c 

4 

4 

4 

19 

d 

17 

1 

e 

15 

17 

0  (13  is 

3 

f 

12 

13 

next  no.*) 

5 

8 

9 

13 

10 

h 

12 

12 

6 

i 

15 

17 

0  (11  is 

2 

j 

15 

16 

next  no.*) 

4 

k 

3 

20 

1 

8 

0 

14 

m 

8 

9 

13 

n 

10 

12 

12  . 

8 

o 

6 

16 

P 

5 

17 

q 

10 

10 

10 

9 

r 

9 

10 

11 

8 

10 

12 

15 

7 

t 

4 

7 

18 

u 

2 

21 

*  no.  =  number 


The  seven  criteria,  in  order  of  importance,  are: 

1.  user  friendly,  menu-driven  screens 

2.  full  CPR/C/SSR  analysis  that  uses  regression  analysis  to  predict 
trends 

3.  easy  to  use  or  can  load  the  software  onto  the  computer  with  no 
problems 

4.  variance  analysis,  with  indicators  of  major  problem  areas 

5.  learn  as  you  use  —  get  to  work  right  away 

6.  needs  to  be  able  to  print  graphics  or  data 
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7.  includes  several  different  Estimate  at  Completion  (EAC)  calculations 
with  a  guide  to  determine  which  EAC  is  most  applicable  for  a  given 
program 

The  two  other  criteria  that  were  tied  for  seventh  place  were: 

8.  ability  to  modify  formulas,  update  charts 

9.  software  is  available  on  personal  computer 

Poliowing  are  the  results  for  each  of  the  three  comparison  methods: 


TABLE  5 


SUMMARY  OP  THE  THREE  COMPARISON  METHOD  RESULTS 
FIRST  COMPARISON  METHOD  SECOND  COMPARISON  METHOD  THIRD  COMPARISON  METHOD 


Order  of 
[raDortance 

Criteria 

Order  of 
Importance 

Criteria 

Order  of 
Importance 

Criteria 

1 

d 

1 

d 

1 

d 

2 

j 

2 

i 

2 

i 

3 

i 

3 

i 

3 

e 

4 

e 

4 

e 

4 

j 

5 

n 

5 

f 

5 

f 

6 

s 

6 

r 

6 

h 

7 

1 

7 

s 

7 

8 

8 

8 

8 

8 

8 

n 

9 

f 

9 

q 

9 

q 

10 

h 

10 

a 

10 

8 

11 

q 

11 

h 

11 

r 

12 

m 

12 

1 

12 

a 

13 

t 

13 

m 

13 

m 

14 

P 

14 

o 

14 

1 

15 

r 

15 

P 

15 

b 

16 

k 

16 

n 

16 

o 

17 

o 

17 

b 

17 

P 

18 

a 

18 

t 

18 

t 

19 

b 

19 

k 

19 

c 

20 

c 

20 

c 

20 

k 

21 

u 

21 

u 

21 

u 

Comparison  of  CPR  Criteria  to  Criteria  from  the  Pour  Studies.  Most 
of  these  criteria  (that  were  determined  to  be  essential  for  CPR  software 
packages  by  the  AFSC  product  divisions)  can  be  grouped  into  one  or  more 
of  the  criteria  categories  from  the  four  studies  that  were  previously 
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reviewed.  The  categories  from  the  four  studies  that  include  the  CPR 
software  criteria  are  highlighted  in  bold  characters  below.  Two  of  the 
CPR  software  criteria  cannot  be  classified  into  any  of  the  criteria 
categories  from  the  four  studies.  They  are!  a  way  to  determine  if  data 
have  been  inputted  correctly  (k)  and  whether  the  software  is  available  on 
a  personal  computer  (q) .  With  regard  to  being  able  to  determine  if  data 
have  been  inputted  correctly,  the  Operations  Research  study  revealed  that 
software  must  be  "user  friendly".  In  this  study,  it  stated  that 
flexibility  of  the  software  was  an  example  of  "user  friendliness"  and  one 
way  to  achieve  this  was  to  avoid  having  the  user  respond  "yes"  or  "no"  as 
to  whether  the  data  entered  were  correct.  Therefore,  a  method  needs  to 
be  found  that  both  will  identify  incorrect  data  and  is  also  "user 
friendly"  at  the  same  time.  As  stated  above,  the  highlighted  areas  in 
Table  6  below  refer  to  the  criteria  categories  from  the  four  studies 
previously  reviewed  which  include  the  criteria  determined  to  be  essential 
for  CPR  software  packages: 


TABLE  6 


COMPARISON  OF  CPR  SOFTWARE  CRITERIA  TO 
SOFTWARE  CRITERIA  FOR  FOUR  DIFFERENT  STUDIES 


Project 

AFSC 

Operations 

Word 

Management 

Software  Quality 

Research 

Processor 

Criteria 

Studv 

Indicators 

Studv 

Studv 

Installation 

X 

Tutorial 

X 

X 

Hands-on  Tutorial 

X 

Data  Bntry  Flexibility 

X 

X 

Reporting 

X 

Customized  Reports 

X 

Quality  of  output 

X 

X 

Interfacing  with  other 

programs  and  computers 

X 

X 

X 

Speed 

X 

X 
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TABLE  6 


COMPARISON  OP  CPR  SOFTWARE  CRITERIA  TO 
SOFTWARE  CRITERIA  FOR  FOUR  DIFFERENT  STUDIES  (Continued) 


Criteria 


Project 

Management 

Study 


AFSC  Operations 

Software  Quality  Research 
Indicators  Study 


Understandable 

documentation  x 

Usable  documentation  x 

Easy  to  learn  (become 

productive  quickly)  x 

Easy  to  Use  -  Screen 

color  mix  x 

Easy  to  Use  ~  Program 

hierarchy  x 

Easy  to  Use  ~  Crrnnd 

presentations  x 

Easy  to  Use  -  help 
function/ useful 
error  messages  x 

Vendor  experience  x 

Vendor  technical 

assistance  x 

Vendor  implementation 

and  support  x 

Vendor  -  training  prgms.  x 

Vendor  -  considers  user 

feedback  x 

Cost  x 

Correctness  (satisfies 
requirements) 

Efficient 

Flexible  (effort  to 


enhance  or  modify  SW) 
Integrity  (ability  to 

control  unauthorized 
access  or  modification 
to  software) 

Maintainable  (effort  reqd. 
to  locate  and  correct 
errors  in  software) 

Portable 

Reliable 

Reusable  (multiple 
applications) 

Testable  (effort  reqd.  to 
ensure  SW  performs 
intended  functions) 
Accurateness 


x 


X 

X 


X 

X 

X 


X 


X 

X 

X 

X 


X 


X 

X 

X 


X 


X 


Word 

Processor 

Study 
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Selecting  the  "Most  Important”  CPR  Software  Criteria 


In  evaluating  the  three  different  comparison  methods  of  the  CPR 
analysis  software  criteria,  it  was  determined  that  the  following  criteria 
consistently  ranked  as  the  four  most  important  criteria  in  evaluating  CPR 
software:  user  friendly,  menu-driven  screens;  variance  analysis; 
regression  analysis;  and  installation.  The  criterion  that  consistently 
ranked  as  number  one  in  all  three  methods  was  user  friendly,  menu-driven 
screens.  The  results  of  the  three  comparison  methods  regarding  the 
features  considered  "most  important”  in  CPR  software  are  as  follows: 


TABLE  7 

RANKINGS  OP  "MOST  IMPORTANT"  CPR  SOFTWARE  CRITERIA 
DETERMINED  FROM  THREB  COMPARISON  METHODS 


- 

Criteria 

Pirst 

Method 

Second 

Method 

Third 

Method 

(d) 

User  friendly,  menu-driven  screens 

1 

1 

1 

(j) 

Variance  analysis 

2 

2 

4 

(i) 

Regression  analysis 

3 

3 

2 

(e) 

Installation 

4 

4 

3 

(f) 

Productivity 

9 

5 

5 

(a) 

Bstimate  at  Completion  (EAC) 

6 

7 

7 

(n) 

Ability  to  modify  formulas  and  update  charts 

5 

16 

8 

(r) 

Interactive  MIS  with  contractor 

15 

6 

11 

(h) 

Ability  to  print  graphics  or  data 

10 

11 

6 

(1) 

Narrative  section 

7 

12 

14 

(g) 

Ability  to  print  on  plotter  or  printers 

8 

8 

10 

(q) 

Software  is  available  on  personal  computer 

11 

9 

9 

(a) 

Help  indexes  that  tutor  on  the  analyses 

18 

10 

12 

(m) 

Documentation 

12 

13 

13 

As  stated  before,  studies  have  shown  that  most  people  can  only 
concentrate  on  seven  plus  or  minus  two  items  at  a  time  before  they  become 
overwhelmed.  Therefore,  the  features  considered  most  important  in  CPR 
analysis  software  will  be  divided  into  the  following  seven  categories: 
User  Friendliness  (Friendly);  Tools;  Modify/Print  Capability  (Mod/Prnt); 
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Interactive  MIS  Capability  (Interact);  Personal  Computer  Capability  (PC); 
Help;  and  Documentation  (Document).  The  specific  features  that  will  be 
included  in  each  category  are: 

Friendly: 

(d)  User  friendly,  menu-driven  screens 

(e)  Installation  (ability  to  load  software  with  no  problems) 

(f)  Productivity  (learn  as  you  use  /  get  to  work  right  away) 

Tools: 

(j)  Variance  analysis 

(i)  Regression  analysis 

(s)  Estimate  at  Completion  (EAC) 

(l)  Narrative  section 
Mod/Prnt: 

(n)  Ability  to  modify  formulas  and  update  charts 
(h)  Ability  to  print  graphics  or  data 

(g)  Ability  to  print  on  plotter  or  printers 
Interact: 

(r)  Interactive  MIS  with  contractor 
PC: 

(q)  Software  is  available  on  personal  computer 

Help: 

(a)  Help  indexes  that  tutor  on  the  analyses  instead  of  referring 
to  a  manual 

Document: 

(m)  Documentation  -  easy  to  understand  and  follow 
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Weights .  In  determining  the  importance  of  each  "moat  important"  CPR 


software  criteria,  an  "average"  approach  will  be  used.  This  approach 
will  be  based  on  how  the  AFSC  product  divisions  ranked  each  criterion 
according  to  the  three  comparison  methods  used  previously  (see  Table  7, 
RANKINGS  OP  "MOST  IMPORTANT"  CPR  SOFTWARE  CRITERIA  DETERMINED  PROM  THREE 
COMPARISON  METHODS).  A  ranking  of  "1"  by  the  product  divisions  received 
a  rating  of  21  points;  "2"  received  a  rating  of  20  points,  etc.  down  to 
where  a  ranking  of  ”21"  received  a  rating  of  1  point.  The  results  are 
provided  in  Table  8  below: 


TABLE  8 

ORDER  OF  IMPORTANCE  OF  "MOST  IMPORTANT"  CPR  SOFTWARE  CRITERIA 
Order  of 


Importance 

Criteria 

Weight 

Category 

1 

(d) 

User  friendly,  menu-driven  screens 

21.0 

Priendly 

2/3 

(i) 

Variance  analysis 

19.333 

Tool  8 

2/3 

(i) 

Regression  analysis 

19.333 

Tools 

4 

(e) 

Installation 

18.333 

Friendly 

5 

(f) 

Productivity 

15.667 

Friendly 

6 

(a) 

Estimate  at  Completion  (EAC) 

15.333 

Tool  8 

7 

(8) 

Ability  to  print  on  plotter  or  printers 

13.333 

Mod/Prnt 

8 

(h) 

Ability  to  print  graphics  or  data 

13.0 

Mod/Prnt 

9/10 

(n) 

Ability  to  modify  formulas  and  update 

charts 

12.333 

Mod/Prnt 

9/10 

(q) 

Software  is  available  on  personal 

computer 

12.333 

PC 

11 

(r) 

Interactive  MIS  with  contractor 

11.333 

Interact 

12 

(1) 

Narrative  section 

11.0 

Tools 

13 

(») 

Documentation 

9.333 

Document 

14 

(a) 

Help  indexes 

8.667 

Help 

Phase  IV:  Method  of  Evaluation 

Expert  Choice,  designed  for  the  IBM  PC  and  compatibles,  is  an  ana¬ 
lytical  hierarchy  model  and  is  based  on  the  theory  developed  by  Dr. 
Thomas  L.  Saaty,  a  professor  of  mathematics  at  the  University  of 
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Pittsburgh.  Expert  Choice  is  decision  support  software  that  facilitates 
a  variety  of  decisions  ranging  from  selecting  a  vacation  to  deciding 
whether  to  build  either  a  hospital  or  nuclear  power  plant.  Expert  Choice 
supports  decision  making  by  coasidering  all  of  the  factors  affecting  the 
decision  as  well  as  how  they  affect  the  decision  at  the  same  time.  The 
human  mind  generally  has  difficulty  doing  this.  Expert  Choice  regards 
the  decision  making  process  as  an  entire  system,  not  as  separate  parts. 

It  considers  both  quantitative  and  qualitative  factors  in  making  a 
decision.  Also,  Expert  Choice  can  accomodate  inconsistencies  that  are 
often  present  in  the  "real”  world.  For  example,  if  someone  prefers  C 
over  D  and  D  over  B,  logic  would  say  that  this  person  would  prefer  C  over 
E.  In  the  "real"  world  this  may  not  necessarily  be  true  be  ;ause  of  other 
factors  that  enter  into  a  decision.  And  Expert  Choice  takes  these  other 
factors  into  consideration  when  making  a  final  decision.  Expert  Choice 
is  user  friendly  in  that  it  allows  the  user  to  incorporate  new  knowledge 
into  the  decision-making  process  as  it  becomes  available. 

Expert  Choice  models  its  decision  making  process  around  an  upside- 
down  tree-like  structure  or  hierarchy  of  the  way  humans  think.  A  single 
goal  is  represented  at  the  top  of  the  structure  and  everything  branches 
down  from  it.  Quantitative  as  well  as  qualitative  criteria  serve  as 
intermediate  levels  on  this  tree.  The  user  decides  which  criteria  to 
model  and  makes  judgments  on  those  criteria.  At  the  bottom  of  the  tree 
are  the  leaves — they  represent  the  alternatives  to  be  chosen.  The  user 
makes  the  final  decision  on  which  alternative  to  select. 

Once  the  goal,  criteria  and  alternatives  have  been  identified, 
judgments  are  made  on  the  criteria.  A  pairwise  comparison  is  conducted 
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on  the  main  criteria  with  respect  to  the  goal.  A  verbal  (for  qualita¬ 
tive)  or  numerical  (for  quantitative)  rating  may  be  used.  With  the 
verbal  rating,  the  terms  "moderately  more",  "strongly  more"  and  "equally 
as"  important  are  used.  With  the  numerical  rating,  numerical  values  or 
their  reciprocals  are  used.  After  the  comparison  on  the  main  criteria  is 
completed,  a  pairwise  comparison  is  conducted  on  the  alternatives  with 
respect  to  each  criterion.  Three  different  types  of  comparison  methods 
are  used  in  Bxpert  Choice.  The;  are  importance,  preference  and  likeli¬ 
hood.  Importance  is  used  when  comparing  a  certain  criterion  with  another 
criterion;  preference  is  used  when  comparing  alternatives;  and  likelihood 
is  used  when  comparing  uncertain  events  (i.e.,  the  probability  of  a 
certain  interest  rate)  (5:2-1  to  2-13). 

Bxpert  Choice  is  unique  in  that  it  offers  the  following  three 
features:  an  inconsistency  ratio,  sensitivity  analysis  and  a  ratings 
utility.  With  regard  to  the  inconsistency  ratio,  a  value  of  0.10  or  less 
is  considered  acceptable.  Bxpert  Choice  will  assist  the  user  in  identi¬ 
fying  his  or  her  most  inconsistent  judgment  and  will  suggest  an  alterna¬ 
tive  judgment.  The  user  is  recommended  not  to  change  their  judgments  in 
order  to  be  more  consistent  unless  he  or  she  thinks  that  those  judgments 
were  initially  inappropriate.  Bxpert  Choice  also  offers  sensitivity 
analysis.  By  changing  judgments  about  the  importance  of  the  criteria, 
the  user  can  determine  how  sensitive  the  final  priorities  for  the  alter¬ 
natives  are  to  those  changes.  The  ratings  utility  is  used  when  there  are 
a  large  number  of  alternatives.  "Large"  refers  to  more  than  seven  alter¬ 
natives  and  Bxpert  Choice  can  handle  up  to  a  few  hundred  alternatives. 
Instead  of  performing  pairwise  comparisons  with  the  alternatives,  the 
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ratings  utility  uses  a  rating  scale  approach.  First,  the  nain  criteria 
are  weighted.  Then,  ratings  "intensities"  (i.e.,  very  expensive,  expen¬ 
sive,  average  and  cheap  may  be  used  to  explain  the  COST  criterion)  are 
entered  and  weighted  for  each  criterion.  For  example,  expensive  is  1.5 
times  as  preferable  as  very  expensive  whereas  cheap  is  4  times  as  prefer¬ 
able  as  very  expensive.  After  this  is  accomplished,  an  incomplete 
spreadsheet  will  appear  with  columns  for  each  criterion.  The  alterna¬ 
tives  are  then  inserted  as  the  rows  of  the  spreadsheet.  Once  this  is 
done,  the  user  assigns  ratings  to  each  alternative  with  respect  to  each 
criterion  using  the  intensity  ratings.  Expert  Choice  will  automatically 
calculate  the  priorities  for  the  alternatives  as  the  user  enters  the 
intensity  ratings.  Even  though  the  intensity  ratings  are  displayed  as 
words,  their  numerical  values  are  used  to  determine  the  total  score  of 
each  alternative.  The  alternatives  can  then  be  sorted  and  ranked  in 
descending  order  based  on  the  total  scores  (6:1-11). 

As  stated  above.  Expert  Choice  can  be  used  in  a  variety  of  areas  of 
decision  making.  The  general  areas  of  application  are  as  follows: 
corporate  executive  decision  making,  corporate  managerial  decision 
making,  small  business,  national  policy  and  public  administration.  The 
more  specific  areas  of  application  contained  within  the  above  categories 
include:  marketing,  finance,  personnel,  engineering,  purchasing  and 
production.  In  addition  to  this.  Expert  Choice  can  be  used  in  many  areas 
of  personal  decision  making  such  as  job  selection,  financial  investments, 
voting  and  purchases  (5:2-6  to  2-8). 
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Phase  V:  Software  Evaluation  and  Comparison  Using  Criteria  Established 


The  four  CPR  analysis  software  packages  were  selected  at  random  in 
determining  the  order  of  evaluation  against  the  criteria.  The  order  of 
evaluation  is  as  follows:  (1)  Performance  Analyzer  (PA);  (2)  CPR-BZ; 

(3)  Contract  Appraisal  System  (CAPPS);  and  (4)  Contractor  Cost  Analysis 
System  (CCAS) . 

Performance  Analyzer  (PA).  Because  Version  2.7  of  the  User  Manual 
was  not  available.  Version  2.6  of  PA's  User  Manual  was  evaluated  while 
Version  2.7  of  PA's  software  was  analyzed. 

Criterion  (d):  User  friendly,  menu-driven  screens.  The  PA's 
menu-driven  screens  are  shown  in  Figure  1,  Performance  Analyzer  Menu 
Tree,  on  page  6  of  the  User  Manual.  The  menu-driven  screens  are 
basically  very  user  friendly  and  easy  to  work  with  except  for  the 
following  observations:  1.)  When  the  PA  software  is  first  accessed  on 
the  computer  screen  and  the  "help"  menu  (FI)  is  entered,  the  software 
displays  "press  any  key  to  continue";  however,  when  "any  key"  is  pressed, 
the  software  does  not  continue;  2.)  In  the  "Automated  Data  Input  Menu," 
when  "*LOTUS"  is  selected,  the  cursor  moves  back  to  "Text";  3.)  It  may  be 
beneficial  to  the  user  if  the  software  included  "FI,  HELP,"  in  the  menu 
at  the  bottom  of  the  screen;  4.)  When  conducting  Functional  Structure 
Maintenance,  the  "ins"  (Insert)  key  allows  the  user  to  add  more  elements. 
The  user  must  use  the  Insert  function  key  and  cannot  perform  the  task  by 
selecting  the  "I"  key;  5.)  Page  33  of  the  User  Manual  states  that  "EAC 
Inputs"  can  be  entered  by  pressing  the  "/"  key.  However,  the  menu 
selection  does  not  include  "EAC  Inputs"  for  Work  Breakdown  Structure 
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(WBS)  Level  1  whereas  it  is  included  for  some  but  not  all  of  the  WBS 
Level  2  and  Level  3  elements;  6.)  With  regard  to  the  Manpower  chart, 
after  the  "Show  Baseline"  option  is  selected,  the  user  must  exit  out  of 
the  graphics  portion  and  reselect  "Manpwr"  before  the  original  Manpower 
chart  is  again  displayed  on  the  screen;  7.)  If  any  changes  are  made  to 
data,  the  user  must  go  to  the  "Contract  Main  Menu"  and  select  the 
"Recalculation"  option  in  order  to  recalculate — it  would  be  both  easier 
and  faster  to  have  a  function  key  perform  this  operation  from  any 
location  in  the  program. 

Criterion  (i);  Variance  analysis.  The  PA  provides  variance 
analysis  for  color  or  monochrome  computer  systems.  The  color  computer 
system  uses  both  arrows  and  colors  in  the  boxes  to  perform  variance 
analysis.  Unless  the  user  either  reads  the  User  Manual  (page  30)  or  the 
Help  screen  (select  FI  for  Help  but  this  is  not  indicated  on  the  menu), 
he/she  may  find  it  difficult  to  distinguish  between  what  the  arrows  and 
colors  symbolize.  For  the  monochrome  computer  system,  the  difference 
between  "normal"  and  "inverse"  is  confusing.  "Inverse"  refers  to  a 
highlighted  box  whereas  "normal"  refers  to  a  blank  box.  Because  the 
variance  analysis  boxes  for  cost,  schedule  and  variance  at  completion  are 
difficult  to  understand  without  referencing  the  Help  menu,  it  may  be 
easier  for  the  user  to  perform  the  analysis  if  a  performance  indicator 
key  (such  as  Table  3  on  page  30  of  the  User  Manual)  is  included  on  the 
same  screen  as  the  variance  analysis  boxes.  Analysis  features  of  PA  are 
described  on  pages  30-32  and  37-38  of  the  User»Manual.  PA  allows  the 
user  to  sort  WBS  and  Functional  levels  by  cost,  schedule  or  variance  at 
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completion  in  order  to  identify  the  major  problem  areas  affecting  each 
situation. 


Criterion  (i):  Regression  analysis.  Linear  regression  is  used 
to  forecast  EAC .  Regression  analysis  can  be  referenced  on  pages  42  and 
88  of  the  User  Manual. 

Criterion  (e^:  Installation.  Installation  procedures  are  on 
page  2  of  the  User  Manual.  The  documentation  states  that  the  user  needs 
to  ensure  that  the  config.sys  file  contains  certain  conmands  but  does  not 
give  step-by-step  instructions  on  how  to  do  this.  Also,  when  loading  PA 
onto  the  computer,  the  software  displays  an  error  message  if  there  is  not 
enough  memory  on  the  computer.  When  this  occurs,  the  user  will  have  to 
delete  another  program  from  the  computer  so  that  PA  can  then  be  loaded 
onto  the  computer  system. 

Criterion  (f):  Productivity.  With  PA,  the  user  can  get  to 
work  right  away  and  is  able  to  learn  as  he/she  uses  the  software.  The 
menus  are  designed  so  that  the  software  is  very  easy  to  use  and  understand. 
As  mentioned  above  in  the  "user  friendly,  menu-driven  screens"  criterion, 
a  "Recalculation"  function  key  may  be  more  oductive  than  accessing  the 
"Contract  Main  Menu"  screen  to  select  the  "Recalculation"  option. 

Criterion  (a):  Estimate  at  Completion  (EAC).  The  PA  includes 
several  different  EAC  calculations  but  does  not  include  a  guide  to 
determine  which  EAC  calculation  to  use  for  a  given  program  and/or 
situation.  The  System  Program  Office  (SPO)  analyst  is  able  to  input 
his/her  own  EAC  along  with  the  method  used  and  a  narrative  rationale.  As 
mentioned  above  in  the  "user  friendly,  menu-driven  screens"  option,  "EAC 
Inputs"  can  be  entered  by  pressing  the  "/"  key.  However,  the  menu 
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selection  does  not  include  "EAC  Inputs"  for  WBS  Level  1  but  does  include 
"EAC  Inputs"  for  some  but  not  all  of  WBS  Level  2  and  Level  3  elements. 

The  User  Manual  references  EAC  on  pages  32,  35,  41-42  and  85-88. 

Criterion  (g):  Ability  to  Print  on  Plotter  or  printers.  Page 
i  of  the  PA  User  Manual  states:  "The  Performance  Analyzer  can  display, 
print,  and  plot  graphs  such  as  analysis  level  trend  data  and  briefing 
class  Program  Management  Review  (PMR)  charts,  required  by  the  Space 
Division  Commander"  (10: i).  Because  a  plotter  was  not  available  to 
analyze  whether  graphics  could  be  plotted,  this  criterion  could  not  be 
evaluated.  However,  the  ability  to  print  on  printers  could  be  evaluated 
and  reports  were  easily  printed  by  using  the  "F7"  key. 

Criterion  (h):  Ability  to  print  graphics  or  data.  Page  36  of 
the  PA  User  Manual  states:  "Graphs  are  displayed  on  the  screen  and  can 
be  output  to  a  printer  or  plotter.  Use  the  'P7'  key  to  print  a  graph  and 
the  'ESC'  key  to  abort  a  print  routine"  (10:36).  This  is  consistent  with 
what  was  discovered  when  the  software  was  evaluated;  however,  the 
"Graphs"  option  only  offers  the  following  selections:  Screen  Plotter 
Exit.  To  print  a  graph,  the  user  must  first  select  the  "Screen"  option 
and  then  press  the  "F7"  key.  This  is  not  obvious  from  viewing  the  menu 
selections.  In  addition  to  this,  printouts  of  the  graphs  are  difficult 
to  read  because  there  are  white  lines  running  through  the  typed  print. 

Page  33  of  the  User  Manual  states:  "Reports  are  displayed  on  the 
screen  and  can  be  output  to  a  printer.  Use  the  *P7'  key  to  print  the 
report  and  the  'ESC'  key  to  abort  a  print  routine"  (10:33).  This  was 
consistent  with  what  was  found  when  the  software  was  evaluated — reports 
were  printed  using  the  "F7"  key,  which  was  very  easy  to  accomplish.  PA 
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offers  numerous  reports  and  graphs — the  reports  are  discussed  on  pages 
33-35  of  the  User  Manual  and  the  graphs  are  described  on  pages  36-39. 

Page  35  of  the  User  Manual  states  that  to  print  the  CPR  report,  "a  wide 
carriage  IBM  printer  set  to  the  condensed  mode  is  required"  (10:35). 
Condensed  print  is  also  required  to  print  the  "PM  Sum"  report  but  this  is 
not  stated  in  the  description  of  the  Program  Manager  Summary  report  on 
page  34  of  the  User  Manual. 

Criterion  (n):  Ability  to  modify  formulas  and  update  charts. 
The  user  is  not  given  the  opportunity  to  modify  formulas  but  as  mentioned 
previously,  the  SPO  analyst  is  able  to  input  his/her  own  EAC  along  with 
the  method  used  and  a  narrative  rationale.  With  regard  to  updating 
charts,  the  user  is  only  able  to  change  the  range  for  some  graphs.  When 
changing  the  range,  it  is  not  easy  to  input  data  and  the  software  does 
not  save  the  changes— it  defaults  back  to  the  previous  data.  Monthly 
data  input  can  be  changed  either  manually  or  via  automation.  Updating 
WBS/Functional  elements  is  also  easy  to  perform  due  to  the  visual  tree¬ 
like  structure  approach  which  PA  uses  (10:17-22). 

Criterion  (q):  Software  is  available  on  personal  computer. 

The  software  packages  were  evaluated  on  personal  computers  (PC)  with  an 
MS-DOS  operating  system.  Performance  Analyzer,  as  well  as  the  other 
three  CPR  software  packages  being  evaluated,  is  available  on  personal 
computer  (PC). 

Criterion  (r):  Interactive  MIS  with  contractor.  Page  i  of  the 

User  Manual  states  the  following: 

The  system  supports  automated  data  transfer  from  contractors  and 
incorporates  automated  interfaces  to  the  Space  Division  ACCI 
Business  Management  Review  (BMR)  System,  the  Defense  Acquisition 
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Executive  System  (DAES)  and  the  Air  Force  Systems  Command  (AFSC) 
Executive  Information  Systems  (BIS).  (10:i) 

With  regard  to  this  criterion,  page  23  of  the  User  Manual  states 
that  PA  performs  error  checks  and  creates  an  error  report  on  the  contrac¬ 
tor's  imported  file. 

Criterion  (1):  Narrative  section.  A  WBS  Narrative  report  is 
available  for  each  WBS  element  and  includes  the  following  information: 
cumulative  performance,  cumulative  cost/schedule  chart,  cost  performance 
indices  chart  and  status  of  element.  PA  also  generates  a  Program  Manager 
Summary  report  (see  pages  34  and  65  of  User  Manual)  and  an  Executive 
Summary  report  (see  pages  34  and  55  of  User  Manual). 

Criterion  (m):  Documentation.  The  PA  User  Manual  is  easy  to 
understand  and  follow  with  the  following  exceptions:  1.)  The  User  Manual 
does  not  include  an  index  but  does  include  a  Table ‘of  Contents; 

2.)  "Thresholds  and  Forecasting  Factors"  on  page  15  is  not  listed  in  the 
Table  of  Contents;  3.)  On  page  iii  of  the  Table  of  Contents,  "Plotter 
Set-up"  and  "Exit"  under  the  "Utilities  Menu"  option  are  not  aligned 
correctly;  4.)  The  Table  of  Contents  lists  the  heading  for  "Contract  & 
WBS/Functional  Maintenance"  on  page  iii  and  the  subheadings  on  page  iv — 
this  is  not  very  easy  to  read;  5.)  The  PA  User  Manual  does  not  spell  out 
the  following  acronyms:  PM  Sum,  MR,  Exec  Sum,  BMR,  CPR,  EAC  (page  v  of 
Table  of  Contents),  OPR  (page  20),  G&A  (pages  17  and  20),  S/L  VAC  (pages 
37  and  39)  and  CAO  (page  40);  6.)  The  following  reports  and  figures  are 
difficult  to  read:  CPR  Formats  1-5  (pages  57-60),  Appendix  2  Formulas 
(pages  81-90)  and  Figure  1  Performance  Analyzer  Menu  Tree  (page  6); 

7.)  Under  "Plot  Summary  PMR  Chart"  on  page  7  of  the  User  Manual,  it 
states  "See  Contract  Information  to  change  the  contract  abbreviation" — 
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this  needs  to  be  more  specific  since  "Initial  Contract  Information"  is 

discussed  on  page  13  and  "Monthly  Contract  Information"  is  discussed  on 

page  24  of  the  User  Manual;  8.)  The  PA  User  Manual  contains  the  following 

misspelled  words:  "Figure  2  Contract  Informantion"  (on  page  13)  instead 

of  Information;  the  first  sentence  of  "Contract  Information"  on  page  13 

states:  "Most  of  the  entries  on  Figure  2”  rather  than  "Most  of  the 

entries  in  Figure  2";  "Table  2  Ini tail  Contract  Data"  (on  page  14) 

instead  of  Initial  Contract  Data;  9.)  The  PA  User  Manual  lists  the 

operation  of  different  "Function  Keys"  separately  on  pages  3,  31-32  and 

42 — a  comprehensive  table  listing  ail  of  the  function  keys  and  their 

respective  operations  may  also  be  helpful  to  the  user.  In  addition  to 

this,  the  User  Manual  for  Version  2.6  and  the  "Foreword"  on  the  software 

for  Version  2.7  list  different  operations  for  certain  function  keys. 

Therefore,  the  function  keys  were  updated  from  Version  2.6  to  Version  2.7 

of  the  Performance  Analyzer..  Examples  of  the  difference  in  the  function 

keys  for  Version  2.6  and  Version  2.7  are  as  follows: 

User  Manual  (Version  2.6) 

Ctrl  +  F4  Weighted  Cost/Schedule 
Ctrl  +  F5  Linear  Regression 
Ctrl  +  F6  User  Performance  Factor 
Ctrl  +  F7  Budget  at  Completion 
Ctrl  ♦  F8  Latest  Revised  Estimate 
Ctrl  +  F9  SPO  EAC  (ICA  Input  Only) 

CPR  Software  (Version  2 . 7)— "Foreword" 

Ctrl  +  F4  Copy  Cur  CPI 
Ctrl  +  F5  Copy  COST  &  SCH 

Ctrl  +  F6  Copy  LINEAR  REG  -  Copies  the  Linear 
Regression  EAC  to  the  SPO  EAC  or 


ICA  EAC. 

Ctrl  +  P7  Copy  PERF  FACTOR 

Ctrl  +  F8  Copy  SPO  EAC  -  Copies  the  SPO  EAC  to 
the  ICA  EAC. 
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Ctrl  +  F9  Copy  NAVSEA  EAC  -  Copies  the  NAVSEA  EAC 
to  the  SPO  EAC  or  ICA  EAC. 

Ctrl  +  F10  Copy  BAC  -  Copies  the  Budget  at 

Completion  to  the  SPO  EAC  or  ICA  EAC. 

Alt  +  F10  Copy  LRE  -  Copies  Latest  Revised 

Estimate  to  the  SPO  BAC  or  ICA  BAC. 

Criterion  (a):  Help  indexes.  PA' 8  help  indexes  tutor  on  the 
analysis  and  can  be  accessed  by  pressing  FI  at  any  location.  The  help 
indexes  provide  thorough,  detailed  explanations  and  contain  basically  the 
same  information  as  the  User  Manual. 

Other  Comments.  Pages  2  and  3  of  the  User  Manual  states  that 
PA  will  eventually  provide  two  types  of  passwords  in  the  network  version: 
one  type  of  password  will  allow  performance  data  to  be  changed  while  the 
other  type  of  password  will  allow  read  only  authorization.  PA  also 
contains  a  Validity  Report  (see  page  34  of  User  Manual)  which  "produces  a 
list  of  analysis  comments  concerning  the  data  validity  and  the  contrac¬ 
tor's  LRE  for  each  WBS  and  Functional  element"  (10:34).  Both  of  these 
features  may  also  enhance  the  productivity  of  the  user  by  providing 
additional  security  and  analysis  techniques. 

CPR-EZ .  CPR-BZ  is  "a  LOTUS  1-2-3  Application"  and  Version  1.0  of 
this  CPR  analysis  software  model  was  evaluated. 

Criterion  (d):  User  friendly,  menu-driven  screens.  CPR-EZ 
uses  menu-driven  screens  which  are  not  very  user  friendly  overall.  Some 
examples  of  these  dif f icult-to-use  screens  are  as  follows:  1.)  In  CPR-EZ, 
the  Escape  (ESC)  key  is  used  to  transfer  control  to  the  user  by 
cancelling  the  menu-driven  feature.  Because  analysts  normally  use  the 
"ESC"  key  to  cancel  information  or  return  to  the  previous  screen  and/or 
menu,  using  the  "ESC"  key  for  some  other  function  may  frustrate  the  user. 
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This  may  occur  i£  the  user  either  forgets  that  the  "ESC"  key  transfers 
control  to  the  user  or  is  unaware  of  this  feature  (i.e.,  does  not  read 
Sections  4.0  or  7.0  of  the  documentation).  A  function  key  that  is  not 
frequently  used  should  be  selected  to  transfer  control  to  the  user. 
Therefore,  using  a  function  key  to  perform  this  operation  would  be 
preferable  to  using  the  "ESC"  key;  2.)  A  major  frustration  with  CPR-EZ's 
menu-driven  screens  is  that  for  whatever  location  of  the  program  the  user 
is  in,  if  he/she  presses  "ESC"  and  then  "Alt-M"  to  restore  the  applicable 
menu  options  for  the  operations  being  performed,  the  main  menu  option 
(DATA  MAINTENANCE  OUTPUT  INSTRUCTIONS  END)  appears.  Therefore,  the 
user  must  start  all  over  again  from  the  main  menu  screen  and  try  to  find 
where  he/she  was  previously  located  in  order  to  complete  the  task  that 
he7 she  was  performing;  3.)  Another  example  of  how  CPR-EZ's  menu-driven 
screens  are  not  user  friendly  is  that  a  myriad  of  screens  must  be 
accessed  before  the  user  is  finally  able  to  print  the  Summary  Report. 
After  the  user  is  able  to  view  the  Summary  Report,  the  next  menu  option 
is;  SUMMARY  REPORT  ALL  OTHER  DETAILS  BACK;  the  menu  option  following 
this  is:  CONTINUE  RECALCULATE;  and,  finally,  the  next  menu  option  is: 
SCREEN  PRINTER  BACK.  This  is  frustrating  to  the  user  if  he/she  wants 
to  print  the  report  immediately  after  viewing  it;  4.)  While  in  the 
"Instructions"  section  of  the  CPR-EZ  or  CPR-DEMO  files,  if  the  user 
selects  the  "Home"  key,  the  program  does  not  return  to  the  beginning  of 
the  information  displayed  when  the  user  first  entered  this  option 
(description  of  model).  Instead,  the  program  returns  to  cell  A1  where  a 
spreadsheet  is  displayed.  This  also  occurs  when  the  user  is  in  location 
BL  through  BR,  where  information  about  printing  graphs  is  located. 
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Following  the  information  about  how  to  print  graphs  is  the  instruction 


"press  RETURN  to  continue";  however,  when  this  is  accomplished,  the 
program  returns  to  cell  Al. 

Criterion  (i);  Variance  analysis.  CPR-EZ  computes  the  cost 
and  schedule  variances  for  all  months.  This  software  package  calculates 
CUM,  6-MO,  3-MO  and  CURR  information  for  each  of  the  following:  CV,  CVS, 
CPI,  SV,  SVZ  and  SPI.  However,  the  software  program  does  not  provide 
indicators  of  major  problem  areas.  The  following  example  is  an  indicator 
of  a  major  problem  area  and  is  available  in  the  CAPPS  program:  "Monthly 
actuals  are  running  at  $17107K.  This  effort  has  an  unfavorable  cost 
variance  of  4  percent.  There  are  indications  that  you've  got  problems 
developing! " 

Criterion  (i):  Regression  analysis.  CPR-EZ  does  not  use 
regression  analysis  to  predict  trends. 

Criterion  (e):  Installation.  CPR-EZ  is  a  LOTUS  1-2-3 
spreadsheet.  It  is  easy  to  load  the  CPR-EZ  files  if  the  user  already  has 
the  LOTUS  program  and  knows  how  to  use  it.  Otherwise,  the  user  must  buy 
either  the  LOTUS  program  or  a  program  that  is  compatible  with  LOTUS  in 
order  to  use  this  spreadsheet.  QUATTRO  and  ENABLB  are  two  software 
programs  that  are  compatible  with  LOTUS.  QUATTRO  was  able  to  load  the 
CPR-EZ  program;  however,  ENABLE  would  not  load  the  CPR-EZ  program  and 
displayed  the  following  message:  "invalid  Spreadsheet  File." 

Criterion  (f):  Productivity.  With  CPR-EZ,  the  user  is  not 
able  to  get  to  work  right  away.  Section  2.0  of  the  documentation 
instructs  the  user  to  retrieve  the  "INSTRUCTIONS"  option  of  the  CPR-EZ 
file  when  using  this  model  for  the  first  time.  CPR-BZ  is  referred  to  as 
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the  CPR  model's  "shell";  the  "INSTRUCTIONS"  option  contains  general 
information  about  how  the  model  works  and  is  not  a  spreadsheet. 

Therefore,  the  user  is  unable  to  get  to  work  right  away. 

Criterion  (s):  Estimate  at  Completion  (EAC).  Section  5.1  of 
the  documentation  lists  the  five  different  EAC  calculations  used  by  CPR-EZ 
(BASIC,  CAPPS,  WEIGHTED,  ACMC  and  LOLLAR) .  However,  there  is  no  guidance 
as  to  which  EAC  formula  to  use  for  a  particular  program  and/or  situation. 
Range  BL186  through  BR197  provides  information  on  the  Weighted  EAC;  range 
BL123  through  BR138  provides  information  on  selecting  an  EAC  method  based 
on  management  reserve  (MR) . 

Criterion  (a):  Ability  to  print  on  plotter  or  printers.  The 
CPR-EZ  program  does  not  print  on  a  plotter.  It  is,  however,  able  to 
print  on  a  printer.  With  CPR-EZ,  data  can  be  printed  using  either  the 
"PRINTER"  menu  option  or  the  "Print  Screen"  function  key. 

Criterion  (h):  Ability  to  print  graphics  or  data.  Section  1.2 
of  the  documentation  states:  "A  graphics  board  is  necessary  for  viewing 
and  printing  graphs."  Additional  information  on  printing  graphs  is 
available  by  accessing  the  "Function  Keys"  option  of  the  "Help  Index" 

(FI)  as  well  as  by  reading  the  information  provided  in  range  CD23  through 
CJ38  and  range  BL3  through  BR21.  Saving  and  printing  graphs  is  not  easy 
to  accomplish.  Range  BL3  through  BR21  tells  the  user  how  to  print  graphs 
but  does  not  give  instructions  on  how  to  get  to  the  LOTUS  main  menu  in 
order  to  choose  the  "PRINTGRAPH"  option  (assumes  the  user  knows  how  to 
use  LOTUS).  Once  the  LOTUS  "PRINTGRAPH"  option  is  set  up  correctly, 
graphs  can  be  printed  on  the  printer.  With  regard  to  being  able  to  print 
data,  the  data  for  individual  WBS  elements  could  not  be  printed.  The 
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"OUTPUT"  option  stated  that  the  user  would  be  able  to  "View  or  print  the 
output";  however,  the  user  is  only  allowed  to  view,  not  print,  data  for 
each  MBS  element.  In  contrast  to  this,  the  software  program  is  capable 
of  printing  the  Sunmary  Report  in  condensed  print  (15  pitch)  so  that  it 
can  be  printed  on  8  1/2"  x  11"  pap'T. 

Criterion  (n):  Ability  to  modify  formulas  and  update  charts. 
Section  3.3  of  the  documentation  states  that  there  are  five  major  inputs 
(BCWS,  BCWP,  ACWP,  BAC  and  LRE)  for  this  software.  There  is  no  limit  to 
the  number  or  characters  that  can  be  entered  in  each  cell  but  only  eight 
digits  will  be  displayed  for  each  input.  The  documentation  also  states 
that  decimals  may  be  entered  but  will  not  be  displayed.  Also,  during 
"INPUT,"  the  cursor  is  confined  to  the  input  area.  When  the  user  tries 
to  edit  data,  the  software  is  set  up  so  that  range  J  through  N  is  frozen. 
This  presents  a  problem  when  the  user  wants  to  view  data  in  range  01 
through  AE120 — as  the  cursor  is  moved  to  the  right,  the  data  in  this 
range  disappears  behind  the  frozen  range  (range  J  through  N) .  Therefore, 
it  is  difficult  for  the  user  to  compare  and/or  view  the  data  when  this 
occurs.  Also,  the  user  is  unable  to  modify  formulas  because  the  cells 
that  contain  the  formulas  are  protected.  With  regard  to  updating  graphs, 
the  user  is  able  to  enter  ranges  for  values  to  be  graphed  and  can  also 
change  the  labels  to  appear  on  the  graphs. 

Criterion  (q);  Software  is  available  on  personal  computer. 

CPR-BZ  Version  1.0  is  available  on  PC. 

Criterion  (r);  Interactive  MIS  with  contractor.  The  CPR-EZ 
"Contract  Analysis  Workshop"  handout  dated  5-6  April  1988  states  that 
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there  is  only  "manual  entry  of  data  -  for  now"  available  with  the  CPR-EZ 
software . 

Criterion  (1):  Narrative  section.  CPR-EZ  generates  a  Summary 
Report  which  includes  X  COMPL  (percent  complete),  BCWS,  BCWP,  ACWP ,  COST 
VAR  X,  SCHED  VAR  X,  BAC,  LRE,  EAC1  (cum  basic  EAC)  and  EAC2  (6-month 
weighted  EAC)  for  each  WBS  element  number.  However,  the  software  program 
does  not  indicate  variance  problem  areas  or  provide  suggestions  for 
further  research.  As  stated  above,  the  Summary  Report  calculates  dollar 
amounts  for  each  WBS  element  number,  including  WBS  level  1.0.  Contrary 
to  this,  the  "INSTRUCTIONS"  option  of  the  CPR-EZ  file  states  the 
following:  "This  model  does  not  calculate  dollar  amounts  at  summary 

levels  (e.g.,  WBS  number  1.0)  based  upon  other  inputs.  In  other  words, 
it  doesn't  sum  the  sub-elements." 

Criterion  (m):  Documentation.  The  pages  of  the  CPR-EZ 
documentation  are  not  numbered.  The  documentation  also  contains  the 
following  typographical  errors:  1.)  In  Section  3.2,  the  first  “ord  of 
the  last  sentence  of  the  first  paragraph,  "do,"  is  not  capitalized; 

2.)  In  Section  3.4,  the  second  line  of  the  column  labeled  "ITEM"  includes 
a  misspelled  word,  "NUMNER."  In  addition  to  this,  the  documentation  does 
not  include  a  Table  of  Contents.  In  Section  5.1,  the  documentation  does 
not  explain  the  theories  behind  the  CAPPS,  ACMC,  LOLLAR,  BASIC  and 
WEIGHTED  EACs.  The  CPR-EZ  documentation  is  easy  to  understand  but  not 
easy  to  follow.  For  example,  the  documentation  does  not  include  all 
features  of  the  program.  The  CPR-EZ  software  contains  three  files:  CPR- 
DEMO,  CPR-DOC  and  CPR-EZ.  However,  only  the  CPR-EZ  file  is  mentioned  in 
the  documentation.  The  CPR-EZ  and  CPR-DEMO  files  are  basically  the  same 
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and  include  menu  options  of  DATA  MAINTENANCE,  OUTPUT,  INSTRUCTIONS  and 
END.  The  CPR-DOC  file  contains  the  same  information  as  the  documentation 
hardcopy  and  includes  the  following  menu  options:  VIEW,  PRINT  and  END. 

One  major  disadvantage  of  the  documentation  is  that  it  does  not  include 
instructions  for  all  operations  that  the  software  can  perform.  As 
mentioned  above,  range  CD23  through  CJ38  and  range  BL3  through  BR21 
contain  instructions  on  how  to  print  graphs.  Likewise,  range  BL123 
through  BR139  and  range  BL186  through  BR197  contain  information  on  the 
EAC;  however,  none  of  this  information  is  included  in  the  documentation. 
Also,  the  "Help  Index"  includes  extensive  information  on  ho*-  to  operate 
LOTUS  1-2-3.  Referencing  this  information  in  the  documentation  may  also 
help  the  analyst  to  learn  how  to  use  the  software  program. 

Criterion  (a):  Help  indexes.  CPR-EZ's  help  facility  does  not 
tutor  on  the  analyses.  However,  it  does  contain  very  thorough  explana¬ 
tions  of  LOTUS  1-2-3  features  such  as  entering,  changing,  adding  and 
deleting  data.  The  PI  function  key  is  the  "help"  key — this  information 
is  not  included  in  the  documentation  or  menu  screens.  The  software 
program  also  refers  the  user  to  the  documentation.  For  example, 

"FORMULAE"  is  a  menu  option  in  the  "OUTPUT"  section  of  the  software 
(select  OUTPUT,  ALL  OTHER  DETAILS,  SELECT  WBS,  SCREEN,  FORMULAB).  When 
the  "FORMULAE"  option  is  selected,  the  software  displays  the  following 
message:  "SEE  DOCUMENTATION." 

Other  Comments.  WBS  information  is  entered  in  the  program 
sequentially;  therefore,  the  relationships  between  the  WBS  elements  are 
difficult  to  visualize.  In  addition  to  this,  Section  6.0  includes  instruc¬ 
tions  on  how  to  locate  certain  information  in  the  software  program.  The 


61 


{Home}  {End}  {Right}  and  {GoTo}  CQ4  {RETURN}  instructions  are  not  accurate. 
{Home}  {End}  {Right}  locates  the  cursor  in  cell  DTI,  where  an  "x"  is 
located.  The  documentation  states  that  performing  these  functions  will 
locate  the  cursor  on  MENUSTOP.  From  cell  DTI,  performing  {End}  {Right} 
locates  the  cursor  in  cell  HI1,  where  MENUSTOP  is  located.  Performing 
{End}  {Right}  again  locates  the  cursor  in  cell  III,  where  MACROSTOP  is 
located.  Performing  the  {GoTo}  CQ4  {RETURN}  operation  locates  the  cursor 
in  cell  CQ4  which  is  the  upper  left  edge  of  the  "CPR-EZ  DETAIL  REPORT." 

The  documentation  states  that  performing  this  operation  will  locate  the 
cursor  in  the  upper  left  edge  of  the  summary  report.  However,  cell  DK4 
is  the  upper  left  edge  of  the  "CPR-EZ  SUMMARY  REPORT." 

Contract  Appraisal  System  (CAPPS).  Version  2.10  of  the  CAPPS 
software  from  Defense  Systems  Management  College  (DSMC)  was  the  third  CPR 
analysis  software  package  to  be  evaluated. 

Criterion  (d};  User  friendly,  menu-driven  screens.  Overall, 
the  menu-driven  screens  for  the  CAPPS  software  are  user  friendly,  with 
the  following  exceptions:  1.)  Entering  the  Contract  Completion  Date  and 
changing  the  Contract  Start  Date  require  a  more  complicated  process  than 
necessary.  The  user  should  be  able  to  access  and  change  this  information 
without  having  to  either  perform  a  variety  of  tasks  or  select  numerous 
menu  options.  Section  5.4.4  on  page  5-7  of  the  User's  Manual  refers  to 
entering  the  "Contract  Completion  Date"  and  Section  5.4.8  on  page  5-8 
refers  to  "Changing  the  Contract  Start  Date";  2.)  If  the  user  selects  the 
[Previous  Screen]  option  while  in  the  "Help"  mode,  a  message  stating 
"can’t  go  back  any  further"  is  displayed  on  the  screen.  However,  when 
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the  [RETURN]  option  was  selected,  the  software  returned  to  the  previous 
screen;  3.)  The  user  can  only  access  certain  graphs  from  specific  menu 
options.  An  example  of  this  is  that  [EAC's]  can  only  be  accessed  while 
in  the  Cost  Performance  Indices  graph  and  [Indices]  can  be  accessed  only 
from  the  Variance  Trends  graph  option.  An  ideal  situation  would  be  if 
the  user  could  access  either  [EAC's]  or  [indices],  etc.  from  any  graphics 
location.  CAPPS  tries  to  compensate  for  this  inability  to  easily  access 
certain  information  by  providing  a  "PAST  TRACK  THROUGH  CAPPS"  chart  on 
page  7-2  of  the  User's  Manual.  Section  7.  PAST  TRACK  on  page  7-1  of  the 
User '8  Manual  states  the  following! 

.  .  .  You  can  get  anywhere  in  CAPPS  by  starting  at  the  Overview 
Screen  or  at  one  of  the  element  Status  Screens.  Prom  any  element 
Status  Screen,  the  graphics  path  is  always  available  (for  that 
element)  by  pressing  the  space  bar  once  to  [Cum  Performance]  and 
pressing  the  Return  key.  (1:7-1) 

Criterion  (j).! _ Variance  analysis.  CAPPS  performs  very 

thorough  variance  analysis  which  includes  indicators  of  major  problem 
areas.  Major  problem  areas  for  each  WBS  element  are  highlighted  in  the 
"Status  of  Element"  screen.  This  information  is  very  easy  to  read  and 
understand  and  contains  a  thorough  analysis  of  each  WBS  element.  Section 
6.4  KEY  ELEMENT  STATUS  on  pages  6-8  and  6-9  of  the  User's  Manual  states 
the  following: 

The  first  section  of  the  screen  states  what  percent  this 
element  is  of  the  total  contract  based  on  the  resources  devoted  to 
it.  This  gives  you  an  idea  of  how  significant  a  problem  the  element 
might  be  in  terms  of  its  impact  on  the  total  contract. 

The  comments  related  to  cost,  schedule,  and  projected  status  in 
the  remaining  three  windows  result  from  a  compilation  of  numerous 
analysis  techniques  applied  to  the  data  elements  in  the  file  for 
that  key  element.  These  comments  [Appendix  A  of  the  CAPPS  User's 
Manual  contains  the  "Rules  for  Status  Screen  Comments . "]  flag 
alarming  or  unusual  trends  using  special  highlighting  techniques  as 
illustrated  by  the  shading  in  Pigure  6-7. 
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The  cost  status  section  provides  the  bottom  line  analysis  based 
on  cost,  schedule  and  technical  performance.  Typical  comments  for 
the  cost  window  warn  you  when  problems  are  apparent  and  identify 
significant  decreases  (increases)  in  the  rate  of  expenditures. 

In  the  schedule  window  you  are  likely  to  see  comments  on 
whether  work  in  process  is  behind  or  ahead  of  schedule,  what  percent 
of  the  total  effort  has  been  completed  to  date,  and  projections  of 
schedule  slip  in  weeks  if  current  performance  trends  continue. 

The  projected  status  window  usually  deals  with  the  feasibility 
of  meeting  the  cost  goals  set  for  the  contract.  Typical  comments 
include  an  evaluation  of  the  feasibility  of  the  contractor's  current 
projections.  (1:6-8  to  6-9) 

As  stated  above,  the  "Key  Elements"  selection  provides  the  user  with 
a  summary  of  the  six  WBS  or  Functional  elements  which  are  causing  the 
most  problems  within  a  program  based  on  cost  variance  percentages.  A 
more  detailed  explanation  on  any  specific  element  is  provided  by  graphic 
presentations.  For  example,  there  is  a  "Variance  Trends  Graph  -  Key 
Element,”  "Variance  Trends  Graph  -  Total  Contract"  and  a  "Cost 
Performance  Indices  Graph."  The  "Variance  Trends  Graph  -  Key  Element" 
graphs  the  cost  variance,  schedule  variance  and  variance  at  completion 
against  time  (horizontal  scale)  and  dollars  (vertical  scale).  The 
"Variance  Trends  Graph  -  Total  Contract"  includes  management  reserve  (MR) 
as  well  as  the  cost,  schedule  and  completion  variances  for  the  contract. 
The  "Cost  Performance  Indices  Graph"  graphs  the  current  and  cumulative 
cost  performance  indices  as  well  as  predicted  cost  performance.  With 
regard  to  predicting  cost  performance,  page  6-21  of  the  User’s  Manual 
states: 

The  cost  performance  Index  To  Go  .  .  .  is  a  measure  of  the  cost 
efficiency  which  would  be  required  to  complete  the  work  within  the 
cost  stated  in  the  Latest  Revised  Estimate.  It  compares  the  planned 
coat  of  the  work  remaining  (BAC-BCWP)  to  the  estimate  to  complete 
(LRE-ACWP).  (1:6-21) 
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Criterion  (i):  Regression  analysis.  The  CAPPS  software  does 


not  use  regression  analysis  to  predict  trends. 

Criterion  (e):  Installation.  While  installing  CAPPS  onto  the 
computer  hard  disk,  a  message  may  appear  stating  that  there  may  be  a 
memory  resident  file  conflict  and  that  the  user  must  remove  any  memory 
resident  programs  from  the  computer  before  running  CAPPS.  Section  5.3  of 
the  User's  Manual  contains  "Installation  Instructions"  for  IBM  and  Zenith 
computers  and  Section  5.6  contains  instructions  for  "Starting  the 
Program."  Section  5.6  contains  incomplete  instructions — for  an  IBM 
version  using  an  BCA  board,  it  states  that  the  user  must  type  "CAPPSE" 

(at  the  system  prompt  for  the  drive  that  the  CAPPS  program  is  in)  to 
bring  up  the  Introductory/Welcome  screen  for  CAPPS;  however,  the  user 
must  type  "CAPPSE.BAT"  to  access  the  screen.  In  addition  to  this, 
instructions  in  Section  5.3  for  how  to  create  and/or  modify  the 
CONFIG.SYS,  GRAPHICS.COM  AND  SCDMP.COM  files  are  not  easy  to  understand 
and  follow. 

Criterion  (f):  Productivity.  With  CAPPS,  the  user  is  able  to 
learn  how  t c  use  the  software  as  he/she  works  on  the  software  program. 
However,  if  a  novice  user  needs  to  access  a  certain  screen  and/or  graph 
immediately,  it  may  be  difficult  for  him/her  to  locate  it  within  a 
reasonable  amount  of  time  without  using  Section  7  of  the  User's  Manual, 
"Fast  Track." 

Criterion  (b):  Estimate  at  Completion  (EAC).  CAPPS  compares 
the  contractor's  Budget  at  Completion  (BAC)  and  Latest  Revised  Estimate 
(LRE)  to  the  following  four  Estimate  at  Completion  (EAC)  values: 
cumulative  cost  performance  (CUM),  the  last  three  months  of  cost 
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performance  (3M0  or  REC),  the  most  recent  month  of  cost  performance  (CUR) 
and  cumulative  cost  and  schedule  performance  (C&S).  Section  6.12, 
"Projected  Completion  Status  -  EAC's,"  of  the  User's  Manual  includes  a 
description  of  how  the  BAC  values  should  be  evaluated  against  BAC  and 
LRE.  The  "EAC  formulas"  are  listed  on  the  computer  under  the  "Explain" 
option  but  are  not  included  in  the  User's  Manual.  "EAC  Techniques"  can 
also  be  accessed  under  the  "Explain"  option — this  screen  very  briefly 
defines  each  type  of  EAC.  The  User's  Manual  refers  to  the  last  three 
months  of  cost  performance  as  "3M0"  while  the  computer  labels  it  as  "REC 
EAC."  CAPPS  does  not  provide  a  guide  to  determine  which  EAC  is  most 
applicable  for  a  given  program  and/or  application. 

Criterion  (a):  Ability  to  print  on  plotter  or  printers.  The 
CAPPS  software  was  designed  to  print  on  printers  but  not  on  the  plotter. 
CAPPS  data  and  graphics  can  be  printed  only  by  using  the  "Print  Screen" 
function  key. 

Criterion  (h):  Ability  to  print  graphics  or  data.  As  stated 
above,  CAPPS  data  and  graphics  can  be  printed  only  by  using  the  "Print 
Screen"  function  key.  With  regard  to  printing  graphs,  pages  5-3  and  5-4 
of  the  User's  Manual  states  that  "the  GRAPHICS.COM  file  allows  the 
contents  of  a  graphics  display  screen  to  be  printed  on  a  compatible 
graphics  printer  when  using  a  color  monitor  and  standard  color  graphics 
card"  (1:5-3  to  5-4).  Page  8-1  of  the  User's  Manual  states  that  "an 
alternative  method  of  printing  the  screen  is  provided  through  the 
PRTSCRN.COM  file  provided  with  CAPPS"  (1:8-1).  The  documentation  also 
states  that  the  screen  can  be  printed  out  with  the  PRTSCRN.COM  file  by 
pressing  the  CONTROL  G  key  combination.  However,  the  file  contained  on 
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the  CAPPS  computer  disk  is  PRTSCRN.EXE,  not  PRTSCRN.COM.  In  addition  to 
this,  graphs  cannot  be  printed  using  either  the  "CTRL  G"  or  the  "Print 
Screen"  key  options.  Page  3-2  of  the  User's  Manual  states  that  CAPPS 
prepares  the  following  four  graphic  analyses:  cumulative  performance, 
variance  trends,  indices  and  projected  completion  status,  or  RAC.  The 
user  is  able  to  either  focus  on  the  most  current  trends  or  view  the 
entire  period  of  performance  of  the  contract  and/or  a  certain  element 
using  the  "Zoom"  and  "Expand"  options  (1:6-12  to  6-14). 

With  regard  to  printing  data,  Section  8,  "Program  Outputs,"  states 
that  the  "PrtSc"  key  on  the  IBM  PC  and  the  P12  key  on  the  Zenith  110  or 
120  will  print  the  information  displayed  on  the  screen.  As  mentioned 
above,  section  8  also  states  that  the  PRTSCRN.COM  file  provided  with 
CAPPS  is  an  alternative  method  of  printing  the  screen.  Data  can  be 
printed  using  either  the  "CTRL  G"  or  the  "Print  Screen"  key  options. 

CAPPS  also  allows  the  user  to  print  a  listing  of  the  information  in  the 
database — this  is  discussed  in  Section  9.6  of  the  User's  Manual. 

Criterion  (n):  Ability  to  modify  formulas  and  update  charts. 
With  CAPPS,  the  user  is  unable  to  modify  formulas.  With  regard  to 
updating  graphs,  the  user  is  able  to  adjust  the  "Variance  Trends  Graph" 
(total  contract  and  key  elements)  for  management  reserve  (MR).  MR  can  be 
viewed  on  the  graph  when  the  user  selects  the  "Adjust"  option.  With 
regard  to  updating  data,  Section  5.5.3  of  the  User’s  Manual,  "Data," 
states  the  following:  "CAPPS  will  work  correctly  only  with  cumulative 
monthly  data.  Also,  CAPPS  does  not  roll  up  the  data.  You  must  enter 


cumulative  data  for  all  elements  at  all  levels"  (1:5-9). 


In  addition  to  this,  the  user  is  only  able  to  evaluate  one  contract 
at  a  time  with  CAPPS.  Pages  9-10  through  9-13  of  the  User's  Manual 
states  that  when  the  user  creates  a  new  contract  database,  CAPPS  erases 
the  current  database.  The  following  screen  prompt  will  also  appear: 

"This  will  erase  the  current  data  base.  If  you  have  not  copied  the  data 
in  the  data  base  to  a  text  file  using  the  SAVB  function,  you  will  lose 
the  data  upon  creating  a  data  base"  (1:9-10).  Therefore,  the  user  must 
prepare  a  separate  disk  for  each  contract  that  he/s-e  monitors. 

As  stated  previously  in  the  "user  friendly,  menu-driven  screens" 
criterion,  the  Contract  Completion  Date  can  only  be  entered  by  using  the 
"Modify"  option  from  the  "Interactive  Update  Menu."  Also,  changing  the 
Contract  Start  Date  requires  more  effort  than  it  should  take  to 
accomplish — this  is  discussed  in  Section  5.4.8  of  the  User's  Manual. 

Section  9.2  of  the  User's  Manual  states  that  with  the  "Interactive 
Update  Menu,"  the  user  can  add  the  next  period  of  data,  correct  data  for 
an  existing  period,  remove  an  existing  WBS  or  Functional  element,  modify 
the  contract  name  or  completion  date  and  create  a  new  contract  database. 
When  the  user  tries  to  edit  data  (select  the  "Utilities,"  "Interactive," 
"Correct,"  and  then  "Edit"  options),  the  software  will  not  allow  the  user 
to  edit  and  displays  the  following  message:  "use  the  '/'  key  to  display 
menu  selections."  Also,  when  the  user  tries  to  add  a  new  element  (select 
the  "Utilities,"  "interactive,"  "Correct"  and  then  "Add  Blement" 
options),  the  software  did  not  save  the  data.  When  the  [Finished]  option 
was  selected,  the  software  displayed  a  message  stating  that  the  database 
was  being  updated;  however,  when  the  database  was  reentered,  the  new 
element  had  not  been  added  to  the  database.  Section  5.4.5  of  the  User's 
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Manual,  "inserting  New  Elements,"  states  that  "when  you  insert  new  WBS 
elements  [in]to  your  data  base,  CAPPS  may  occasionally  put  the  element 
out  of  sequence,  or  may  duplicate  the  element"  (1:5-7).  The  User's 
Manual  does  give  the  user  instructions  on  how  to  correct  this  problem; 
however,  it  is  a  lengthy  process  and  requires  more  effort  than  necessary 
to  perform  this  function. 

Criterion  (q):  Software  is  available  on  personal  computer. 

The  CAPPS  software  is  available  on  PC. 

Criterion  (r):  Interactive  MIS  with  contractor.  CAPPS  does 
not  provide  an  interactive  Management  Information  System  (MIS)  with  the 
contractor  in  which  CPR  data  come  in  electronically  and  automatically  go 
into  a  spreadsheet  to  be  analyzed.  However,  CAPPS  does  provide  for 
automated  data  submission  from  the  contractor  (see  Appendix  G  of  the 
User's  Manual).  Figure  9-9,  on  page  9-13  of  the  User's  Manual,  states 
the  following:  "In  Batch  mode,  preformatted  information  is  automatically 
read  into  CAPPS,  element  by  element.  Batch  mode  was  designed  for  large 
amounts  of  data  which  have  been  prepared  'off-line'  (eg.  by  the  contrac¬ 
tor)"  (1:9-13).  Section  9.4  of  the  User's  Manual,  "Batch  Update,"  states 
that  if  a  bad  data  message  appears  while  the  data  are  being  read  in, 

CAPPS  will  usually  pass  on  to  the  next  item  of  data  (1:9-16).  With 
regard  to  this,  CAPPS  needs  to  be  able  to  provide  the  user  with  the 
location  of  the  "bad  data." 

Criterion  (1):  Narrative  section.  CAPPS  does  not  provide  a 
narrative  section  which  sums  all  data,  trends  and  variance  problems  and 
does  not  provide  suggestions  for  further  research.  Page  2-2  of  the 
User’s  Manual  states:  "The  narrative  explanations  are  not  part  of  CAPPS' 
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analysis;  neither  are  CPR  Formats  3  and  4.  These  Formats  should  not  be 
ignored,  however,  since  they  provide  important  information  for  program 
management"  (1:2-2). 

Criterion  (m):  Documentation.  The  CAPPS  User's  Manual  is 
basically  easy  to  understand  and  follow;  however,  there  are  several  areas 
of  the  User's  Manual  which  do  not  include  step-by-step  instructions  on 
how  to  accomplish  certain  tasks.  Some  of  these  areas  include  the 
following:  1.)  Section  5.3,  "Installation  Instructions";  2.)  Section 
5.4.8,  "Changing  the  Contract  Start  Date";  3.)  Section  5.4.5,  "Inserting 
New  Elements";  4.)  Section  9.4,  "Batch  Update";  and  5.)  Section  9.3, 
"Discussion  of  Interactive  versus  Batch  Update."  Other  observations 
about  the  CAPPS  User's  Manual  include:  6.)  Figures  9-5  and  9-6  (on  pages 
9-5  and  9-6)  are  difficult  to  read;  7.)  Page  6-22  refers  to  the  last 
three  months  of  cost  performance  as  "3M0"  while  the  software  labels  it  as 
"REC  EAC";  8.)  The  User's  Manual  does  not  list  the  formulas  for  the 
calculations  performed  by  the  CAPPS  software;  9.)  The  computer  software 
lists  "Commands"  for  the  function  keys  in  the  "MAIN  HELP  MENU";  however, 
this  information  was  not  available  in  the  User's  Manual;  and  10.)  The 
User's  Manual  and  the  computer  software  contain  conflicting  information 
in  describing  the  line  type  and  color  of  the  various  performance 
measurement  data  elements  for  the  following  graphs:  "Cumulative 
Performance,"  "Cost  Performance  Indices"  and  "Variance  Trends."  Table  9 
below  shows  a  comparison  of  the  information  in  the  User's  Manual  and  on 
the  computer  software. 
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TABLE  9 


COMPARISON  OF  PERFORMANCE  MEASUREMENT  INFORMATION 
IN  CAPPS  USER'S  MANUAL  vs  COMPUTER  SOFTWARE 

Cumulative  Performance  Graph 

CAPPS 
Software 


User  s  Manual 
(Page  6-11) 


BCWP  is  a  white  dotted  line 
BCWS  is  a  long  dashed  red  line 
ACWP  is  a  solid  blue  line 
BAC  is  a  dotted  blue  line 

LRE  is  a  short  dashed  red  line 
(Zenith)  or  blue  line  (IBM) 


BCWP  is  a  long  dashed  white  line  —  — 

BCWS  is  a  short  dashed  red  line  -  -  -  - 
ACWP  is  a  solid  blue  line 
BAC  is  a  long  dashed/short  dashed  white 
line  —  - 

LRE  is  a  long  dashed/short  dashed/long 
dashed  red  line  on  the  IBM  —  -  — 


Variance  Trends  -  Total  Contract  Graph  ** 


User's  Manual 
(Page  6-18) 


CAPPS 
?f tware 


Management  Reserve  (MR)  is  a 
blue  dotted  line 


MR  is  a  long  dashed/short  dashed  white 
line  —  - 


**  The  Variance  Trends  -  Total  Contract  Graph  is  very  difficult  to  read — 
there  is  a  solid  white  line  on  this  graph  but  no  caption  to  identify  this 
line . 
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TABLE  9 


COMPARISON  OF  PERFORMANCE  MEASUREMENT  INFORMATION 
IN  CAPPS  USER'S  MANUAL  vs  COMPUTER  SOFTWARE 

(Continued) 


Variance 

User's  Manual 
(Page  6-16) 

CV  is  a  solid  blue  line  i 

SV  is  a  dashed  red  line  i 

VAC  is  a  dotted  white  line  ! 


Trends  Graph 

CAPPS 

Software 

CV  is  a  solid  blue  line 
SV  is  a  short  dashed  red  line 
VAC  is  a  long  dashed  white  line 


Some  positive  observations  about  the  CAPPS  User's  Manual  include: 

1.)  Section  2  contains  a  thorough  explanation  of  "The  Performance  Measure¬ 
ment  Process";  2.)  Page  7-2  contains  a  "Fast  Track  Through  CAPPS"  chart 
which  is  designed  to  assist  the  user  in  going  from  one  screen  to  another; 
and  3.)  The  User's  Manual  is  easy  to  read  because  it  contains  high-quality 
print  and  the  documentation  is  placed  in  a  notebook. 

Criterion  (a):  Help  indexes.  CAPPS'  contains  "Help"  and 
"Explain"  indexes  which  tutor  on  the  analyses  so  that  the  user  will  not 
have  to  refer  to  a  manual.  However,  with  CAPPS,  the  user  may  have  to 
refer  to  Section  7  of  the  User's  Manual,  "Fast  Track,"  in  order  to  access 
certain  screens  and/or  graphs  quickly.  Appendix  B  of  the  User's  Manual 
describes  the  User  Assistance  function.  The  User  Assistance  function  is 
composed  of  the  "Help"  and  "Explain"  options.  "Explain"  provides  the 
user  with  information  about  performance  measurement  terminology  and 
techniques;  "Help"  provides  information  about  hardware  and  software 
operation.  "Help"  can  be  accessed  from  any  location  with  the  "?"  key. 
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The  "Explain"  screens  contain  very  thorough,  detailed  definitions  of 
BCWP ,  BCWS ,  ACWP,  BAC  and  LRE. 

Other  Comnents.  The  CAPPS  software  is  available  in  three 
versions:  a  color  graphics  version,  a  tabular  version  and  a  monochro¬ 
matic  version.  CAPPS  also  performs  logic  and  consistency  checks  during 
its  analyses  of  contract  performance  data.  Also,  a  "DSMC/PMSS  Software 
Modules  Update"  dated  July  1990  states  that  CAPPS  software  is  currently 
being  enhanced  to  include  VGA  display  capabilities  and  to  allow  use  of 
laser  printers.  As  stated  previously,  "DSMC"  is  the  acronym  for  Defense 
Systems  Management  College  and  "PMSS"  is  Program  Manager's  Support  System. 


Contract  Cost  Analysis  System  (CCAS).  Version  1.1  of  the  CCAS  CPR 
analysis  software  package  was  prepared  by  Management  Consulting  & 
Research,  Inc.  (MCR)  in  Palls  Church,  VA,  for  the  U.S.  Air  Force  Cost 


Center. 


tcreens .  CCAS ' 


menu-driven  screens  are  not  user  friendly.  Some  examples  are  as  follows: 
1.)  It  is  difficult  to  save  information  with  the  CCAS  software.  When 
trying  to  save  data  that  have  been  added  and/or  edited  in  CCAS,  the  user 
must  either  exit  and  then  reenter  CCAS  or  select  the  "Clean  up  Files" 
option  from  the  "Utilities"  menu.  Calculations  may  be  performed  faster 
and  easier  if  the  CCAS  software  allowed  a  function  key  to  save  updated 
information;  2.)  The  program  completion  date  for  the  program  that  was 


developed  on  the  software  was  "JUN  93."  When  trying  to  obtain 
information  for  the  "PROJECTED  BCWS  ENTRY  or  MODIFICATION"  prompt,  a 
message  stating  "Date  can't  be  later  than  the  completion  date"  was 
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displayed  for  the  following  dates  that  were  entered:  JUN81,  JUN93  and 
JUN91;  3.)  The  CCAS  software  does  not  allow  the  user  to  access  previous 
screens  or  menu  options  from  all  screens.  Also,  pressing  the  "Esc"  key 
enables  the  user  to  return  to  the  previous  menu  for  only  some  of  the 
screens  in  CCAS;  4.)  While  in  the  "Add/Edit  Milestones"  screen,  the 
cursor  must  be  within  the  "Add/Edit  Milestones"  block  before  pressing 
"<ESC>  to  Remove  Menu"  will  be  executed;  in  other  words,  the  cursor  must 
be  located  in  a  certain  portion  of  the  screen  before  any  operations  can 
be  performed;  5.)  When  the  "PI"  key  is  selected  to  perform  the  "Help" 
function,  the  software  displays  either  one  of  two  messages — the  first 
message  may  be  an  error  reading  and  the  software  asks  the  user  if  he/she 
wants  to  ’QUIT";  the  second  message  states  that  "Help  is  currently  not 
available  at  this  point."  The  "Tutorial"  option  also  displays  the 
following  message:  "assist  is  not  available  at  this  point";  6.)  When 
selecting  reports  to  print,  the  CCAS  software  allows  the  user  to  select 
"<A>  for  all  reports"  and  "<C>  tc  clear  choices."  Pressing  "C"  will 
deselect  all  choices  and  the  user  must  reenter  the  number(s)  for  the 
report(s)  that  he/she  wants  to  print.  The  "<C>  to  clear  choices"  option 
could  be  more  user  friendly  if  it  only  deselected  the  report(s)  that  the 
user  did  not  want  to  print  instead  of  deselecting  all  of  the  reports; 

7.)  The  CCAS  software  is  inconsistent  in  using  a  certain  function  key  to 
perform  the  same  or  similar  tasks  for  various  screens.  Por  example,  the 
"BCWS  Projections  Screen"  requires  the  user  to  press  "<CR>  to  exit" 
(14:111-48).  However,  the  "Add/Edit  Milestones  Menu"  requires  the  user 
to  press  "<ESC>  to  Remove  Menu"  (14:111-50);  8.)  When  in  the  "Historical 
Performance  Data"  option  for  "Add/Edit  Data,"  the  software  states  that 
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the  analyst  can  use  "<Pg  Up>/<Pg  Dn>/<Arrows>  to  position."  However,  the 
"<Pg  lm>"  key  does  not  perform  this  operation;  9.)  When  the  "Perform 
Calculations"  option  is  selected  from  the  "Utilities"  menu,  the  software 
states  that  there  is  a  "Database  error  numeric  overflow"  and  asks  the 
user  if  he/she  wants  to  "QUIT";  and  10.)  The  CCAS  software  is 
inconsistent  in  labelling  menu  screens  and  the  reports  associated  with 
those  menu  screens.  Following  is  a  comparison  of  the  "Report  Options" 
menu  screen  (14:111-12)  and  the  actual  titles  of  those  reports  (14:111-16 
to  III-21): 


"REPORT  OPTIONS" 

Menu  Screen _ 

Contract  Status  Sumnary 
Dollar  Variances 
Percents  and  Indices 
Incremental  Status 
Estimates  at  Completion 
Milestones 


Actual  Title 
_ of  Report 

Contract  Status  Sumnary 
Dollar  Variance 
Percentage  Variance 
Current  Period  Data 
Projected  Status 
Milestones 


Criterion  (i):  Variance  analysis.  The  CCAS  CPR  analysis 
software  performs  variance  analysis  but  does  not  provide  indicators  of 
major  problem  areas.  Page  III-5  of  the  Users  Manual  lists  the  following 
types  of  analysis  techniques  performed  on  the  data  and  available  under 
the  "Screen  Displays"  option  of  the  "Detailed  Analysis"  menu: 

"Cumulative  Data,"  "Variance  Percents  &  Indices,"  "Incremental  Data,"  . 
"Projected  Status"  and  "Milestone  Status." 

Criterion  (i):  Regression  analysis.  The  CCAS  software  does 
not  use  regression  analysis  to  predict  trends. 

Criterion  (e):  Installation.  CCAS  was  installed  by 
downloading  the  CCAS  program  from  the  Cost  Center  Bulletin  Board  via  a 
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modem.  The  phone  number  of  the  Cost  Center  Bulletin  Board  is 
1-800-344-3602.  In  downloading  the  CCAS  program,  the  user  will  need  to 
download  the  ARC522.BXE  and  the  CCAS. ARC  files  from  the  Bulletin  Board. 
The  user  may  also  want  to  print  out  instructions  for  "How  to  Use 
Arc522.EXE"  which  can  be  obtained  from  the  "Bulletins"  option  of  the 
Bulletin  Board.  The  ARC522.EXE  and  CCAS. ARC  files  can  be  accessed  under 
the  "Piles"  option  of  the  Bulletin  Board.  After  the  "Piles"  option  is 
selected,  the  user  then  performs  the  "Download"  operation  and  usee  the 
"Page  Down"  function  key  to  begin  the  downloading  process.  Downloading 
files  from  the  Cost  Center  Bulletin  Board  is  not  a  difficult  task. 
Instructions  for  dual  floppy  and  hard  disk  installation  can  be  obtained 
in  Section  II. B.  of  the  Users  Manual,  "Getting  Started." 

Criterion  (f):  Productivity.  With  CCAS,  the  user  is  unable  to 
get  to  work  right  away.  This  is  due  mainly  to  the  following  error 
message  displayed  on  the  screen  when  the  user  selects  the  "Detailed 
Analysis"  option:  "proc:  DBFOPN  line.*204  DOS  error  3,  C:\A10\C0NTRACT." 
When  this  error  message  is  displayed,  the  user  needs  to  enter  "Program 
Information,"  "Contract  Information"  and  "Historical  Performance  Data" 
under  the  "Add/Bdit  Data"  option.  However,  the  user  may  not  be  able  to 
determine  the  cause  of  the  problem  because  he/she  is  unable  to  decipher 
the  above  error  message.  Therefore,  when  using  CCAS,  productivity  may  be 
improved  by  displaying  comprehensible  error  messages.  As  mentioned 
previously,  the  "Perform  Calculations"  option  of  the  "Utilities"  menu 
must  be  accessed  in  order  to  perform  calculations  on  updated  information. 
If  a  function  key  was  used  to  perform  this  operation,  increased 
productivity  may  result  (i.e.,  the  user  would  be  able  to  access  the 
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function  key  more  easily  than  the  "Utilities"  menu,  resulting  in 
calculations  being  performed  in  less  time  than  previously  required). 

Also,  with  the  present  version  of  the  CCAS  program,  the  user  may  assume 
that  the  software  automatically  saves  and  updates  changes  made  to  data. 
Therefore,  the  software  needs  to  display  a  message  telling  the  user  how 
to  update  and  save  changes  he/she  has  made  to  the  data. 

Criterion  (s):  Estimate  at  Completion  (RAC).  CCAS  performs 
four  different  EAC  calculations  but  does  not  include  a  guide  to  help  the 
user  determine  which  EAC  is  most  applicable  for  a  given  program  and/or 
situation.  The  four  BAC  calculations  are:  Method  1  -  Cumulative 
Performance,  Method  2  -  Recent  (6  Month)  Performance,  Method  3  -  Current 
Period  Performance  and  Method  4  -  Cost  &  Schedule  Performance  with 
Weightings.  Exhibit  III-22  on  page  III-28  of  the  Users  Manual,  "EAC 
Selection  Grid,"  lists  the  following  categories  that  comprise  each  of  the 
four  EAC  methods:  combination  methodology,  months  of  history,  cost 
weighting  factor  and  schedule  weighting  factor.  In  addition  to  this. 
Appendix  A  (of  the  CCAS  Users  Manual)  contains  a  thorough  description  of 
the  EAC  calculations. 

Criterion  (g):  Ability  to  print  on  plotter  or  printers.  As 
stated  previously,  because  the  capability  to  print  on  a  plotter  was  not 
available,  this  criterion  could  not  be  evaluated.  However,  page  III-15 
of  the  Users  Manual  states  that  the  "Hard-Copy  Graphics"  option  will  plot 
the  following  graphs:  "Cumulative  Performance,"  "Variance  Trends," 
"Performance  Indices"  and  "Estimates  at  Completion."  Page  II-l  (of  the 
Users  Manual)  states  that  one  of  the  minimum  hardware  requirements  to 
operate  CCAS  is  a  HP7475A  or  Western  Graph-Tech  MP02300  plotter.  With 
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regard  to  being  able  to  print  on  printers,  CCAS  has  the  capability  to 
print  its  reports  in  condensed  print.  The  reports  are  also  printed  in  a 
professional  briefing  class  format.  The  menu  screens  to  print  the 
reports  are  easy  to  use;  however,  as  mentioned  previously,  the  "<C>  to 
clear  choices"  option  clears  all  of  the  choices  when  selected.  The  "<C>" 
option  could  be  more  user  friendly  if  it  only  deselected  certain  choices 
instead  of  deselecting  all  of  the  reports  to  print. 

Criterion  (h):  Ability  to  print  graphics  or  data.  Exhibits 
III-17  through  III-21  on  pages  III-22  through  III-26  of  the  Users  Manual 
display  the  graphics  options  available  with  CCAS.  CCAS  does  not  allow 
the  user  to  view  the  graphs  on  the  computer  screen  before  plotting  them. 
The  types  of  reports  available  with  CCAS  and  the  menu  screens  describing 
how  to  print  those  reports  are  discussed  in  the  Users  Manual  on  pages 
III-8  through  III-15.  Printouts  of  the  various  reports  are  available  on 
pages  III-16  through  III-21.  The  CCAS  software  allows  the  user  to  print 
in  condensed  print.  However,  as  stated  above,  the  "<C>  to  clear  choices" 
option  when  selecting  reports  to  print  is  not  user  friendly.  As  stated 
on  page  III-8  of  the  Users  Manual,  "there  are  two  Program  Reports  and  six 
Detailed  Reports  that  can  be  selected"  (14:111-8).  There  is  also  a 
"Contract  Status  Summary"  option  under  both  Program  Reports  and  Detailed 
Reports.  When  developing  the  program  and  contract  information  to  be 
analyzed  by  this  software,  the  user  created  only  one  contract  for  this 
program.  However,  conflicting  information  was  generated  for  each 
"Contract  Status  Summary"  option  under  the  two  types  of  reports  (Program 
Reports  and  Detailed  Reports).  When  printing  the  various  reports,  the 
"<A>  for  all  reports"  option  was  selected.  In  the  Program  Reports 
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option,  the  "Contract  Status  Summary"  report  was  printed  out  once  whereas 
the  "Contract  Indices"  report  was  printed  out  twice.  In  the  Detailed 
Reports  option,  the  "Contract  Status  Summary"  report  can  only  be  printed 
using  the  "Select  Specific  Contract"  option.  The  remainder  of  the 
reports  in  the  Detailed  Reports  section  printed  out  continuously  under 
the  "All  Contracts"  option. 

Criterion  (n):  Ability  to  modify  formulas  and  update  charts. 
Section  III.C.  of  the  Users  Manual,  "Add/Edit  Data,"  describes  the 
various  options  available  in  which  data  can  be  updated;  Exhibit  III-23  of 
the  Users  Manual,  "Add/Edit  Menu,"  displays  these  options.  When  trying 
to  insert  a  report  date  in  the  "Historical  Performance  Data"  entry  screen 
(select  "Add/Edit  Data"  and  then  "Historical  Performance  Data"),  if  the 
user  does  not  either  select  "<Pg  Up>  to  position"  or  perform  the  "Clean 
up  Files"  option  in  the  "Utilities"  menu  prior  to  inserting  the  new 
report  date,  the  software  may  respond  with  the  following  message:  "That 
date  already  exists!  Press  RETURN  &  Enter  a  new  date."  As  mentioned 
before,  in  the  "Historical  Performance  Data"  screen  the  software  st?tes 
that  either  the  "<Arrows>,  <Pg  Up>  or  <Pg  Dn>"  keys  can  be  used  "to 
position";  however,  the  "<Pg  Dn>"  function  key  does  not  perform  this 
operation. 

With  respect  to  the  Over  Target  Baseline  (OTB)  Adjustment,  page  HI- 
29  of  the  Users  Manual  states  that  CCAS  allows  the  user  to  either  recalcu¬ 
late  the  "performance  measurements  against  an  overtarget  baseline  or  a 
baseline  measured  against  contract  values"  (14:111-29).  The  Users  Manual 
also  states  that  CCAS  indicates  which  calculations  have  been  performed  by 
positioning  either  an  "A"  (for  adjusted  values)  or  an  "R"  (for  report 


79 


values)  "below  the  contractors  name  in  the  header  of  screen  reports  and 
hard  copy  printed  reports"  (14:111-29).  However,  when  the  values  are 
"adjusted,"  the  software  displays  a  message  stating  that  there  is  a 
database  error  (due  to  numeric  overflow)  and  asks  the  user  if  he/she 
wants  to  "QUIT."  Also,  contrary  to  the  above  statement  in  the  Users 
Manual  which  stated  that  either  an  "A"  or  an  "R"  would  be  printed  on  hard 
copy  reports  below  the  contractor's  name,  this  was  not  accomplished. 

In  evaluating  the  "EAC  Selection  Grid,"  page  III-27  of  the  Users 
Manual  states  that  the  Combination  Methodology  column  is  "fixed  and 
cannot  be  changed  by  the  user"  (14:111-27).  However,  this  column  can  be 
changed  by  the  user  and  even  though  the  software  will  not  save  the 
changes  made  in  this  column,  it  will  perform  calculations  based  on  the 
updated  symbol  for  EAC  1,  3  and  4.  The  software  does  not  perform  calcu¬ 
lations  on  the  updated  symbol  for  EAC  2.  The  Users  Manual  states  that 
the  Months  of  History,  Cost  Weighting  Factor  and  Schedule  Weighting 
Factor  columns  can  be  updated  by  the  user — this  was  found  to  be  accurate. 

With  regard  to  being  able  to  update  graphs,  page  III-53  of  the  Users 
Manual  states  that  the  user  is  able  to  rescale  the  data.  "This  rescaling 
provides  the  vertical  scale  for  the  hard  copy  graphic  plots”  (14:111-53). 
However,  when  the  "Autoscale  Plot  Data"  option  is  selected  from  the 
"Utilities"  menu,  the  software  does  not  allow  the  user  to  input  data. 

Criterion  (q):  Software  is  available  on  personal  computer. 
Version  1.1  of  the  CCAS  CPR  analysis  software  package  is  available  on  PC. 

Criterion  (r):  Interactive  MIS  with  contractor.  CCAS  has  the 
capability  to  provide  an  interactive  MIS  with  the  contractor  in  which  CPR 
data  come  in  electronically  and  automatically  go  into  a  spreadsheet  to  be 
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analyzed;  however,  this  capability  is  available  only  for  Defense 
Acquisition  Executive  Sunmary  (DAES)  programs.  Data  can  also  be  updated 
with  CCAS  from  a  DAES  transfer  disk  for  Defense  Acquisition  Executive 
Summary  (DAES)  programs  and  a  nonautomated  entry  process  for  non-DAES 
programs.  Pages  III-30  through  III-36  of  the  Users  Manual  discuss  how 
DAES  information  is  updated  on  CCAS. 

Criterion  (1):  Narrative  section.  CCAS  does  not  include  a 
narrative  section  which  sums  all  data,  trends  and  variance  problems  or 
provides  suggestions  for  further  research.  The  "Top  Level,  Summary 
Analysis"  option,  designed  to  "provide  the  user  with  the  ability  to 
quickly  focus  on  a  small  set  of  problem  programs,"  will  be  implemented  in 
later  versions  of  the  software  (14:111-1). 

Criterion  (m):  Documentation.  The  CCAS  software  Users  Manual 
is  very  easy  to  understand  and  follow.  A  Users  Manual  for  "Version  1.0 
of  CCAS"  was  provided;  however,  the  software  disks  were  labeled  "CCAS 
Version  1.1."  In  addition  to  this,  the  subheading  titles  and  their  page 
numbers  are  not  listed  in  the  Table  of  Contents.  The  Users  Manual 
contains  thorough  explanations  and  includes  visual  displays  of  the 
computer  screens,  printed  reports  and  graphs.  Even  though  the  Users 
Manual  is,  for  the  most  part,  very  easy  to  understand  and  follow,  the  0TB 
Adjustment  explanation  (on  page  III-29)  is  somewhat  difficult  to  follow. 
Also,  with  regard  to  the  "Hard  Disk  Installation"  instructions  (on  page 
II-3),  step-by-step  instructions  may  be  needed  to  assist  the  user  in 
determining  if  the  CONFIG.SYS  file  contains  the  appropriate  operations. 
The  documentation  also  contains  minor  typographical  errors — an  example  of 
this  is  on  page  A-2  which  states:  "The  system  can  develop  a  variety  of 
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Performance  Factors.  These  can  range  from  Method  1,  Cumulative 
Performance  to  Method  3,  Current  Period  Performance"  (14:A-2).  However, 
there  are  four  instead  of  three  EAC  methods .  The  CCAS  Users  Manual  also 
omits  some  information  that  may  be  very  valuable  to  the  user.  For 
example,  in  the  "Historical  Performance  Data"  screen,  the  user  may  need 
to  select  the  "<Pg  Up>"  key  "to  position"  before  he/she  tries  to  insert  a 
new  report  date.  Otherwise,  a  message  unrelated  to  the  cause  of  the 
problem  may  be  displayed  (i.e.,  When  the  message  "That  date  already 
exists!  Press  RETURN  &  Enter  a  new  date."  is  displayed,  the  problem  may 
be  that  the  software  has  not  ordered  the  dates  in  a  logical  sequence 
rather  than  the  user  has  entered  a  date  that  already  exists.).  On  the 
positive  side,  Appendix  A  of  the  Users  Manual  lists  all  formulas  used  in 
CCAS. 

Criterion  (a):  Help  indexes.  CCAS  does  not  include  help 
indexes  that  tutor  on  the  analyses.  Page  1-1  of  the  Users  Manual  states 
the  following:  "CCAS  contains  a  complete,  on-line  help  function  as  well 
as  a  tutorial  function  to  support  the  user"  (14:1-1).  However,  when 
either  the  help  or  tutorial  functions  are  accessed,  the  screen  displays 
the  following  messages,  respectively:  "help  is  not  available  at  this 
point"  or  "assist  is  not  available  at  this  point." 

The  results  for  evaluating  the  four  PC-based  CPR  analysis  software 
packages  against  the  fourteen  "most  important"  user-established  criteria 
are  listed  in  Table  10  below.  Evaluation  of  the  CPR  analysis  software 
packages  was  based  on  the  descriptions  of  the  user-established  criteria. 
For  example,  the  software  packages  were  evaluated  against  the  documenta¬ 
tion  criterion  of  being  easy  to  understand  and  follow,  not  that  the 
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software  package  included  documentation.  An  "x"  in  Table  10  signifies 
that  the  CPR  analysis  software  package  includes  the  specified  user- 
established  criterion;  however,  it  does  not  signify  the  degree  to  which 
the  software  package  meets  the  criterion.  A  description  of  each 
criterion  is  available  in  Phase  II  of  Chapter  III. 

Chapter  III  consisted  of  selecting  the  criteria  used  to  evaluate  the 
four  CPR  analysis  software  packages  in  this  study  and  then  evaluating  the 
software  packages  against  each  of  the  fourteen  "most  important"  user- 
established  criteria.  In  Chapter  IV,  "the  ideal"  CPR  analysis  software 
package  will  be  selected  using  a  decision  support  software  program. 

Expert  Choice. 
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TABLE  10 


SUMMARY  FOR 

EVALUATION  OF  CPR  ANALYSIS  SOFTWARE 
PROGRAMS  AGAINST  USER-ESTABLISHED  CRITERIA 


Criterion 

CPR  Analysis 

Software 

ProRram 

Symbol 

Description 

PA 

CPR-EZ 

CAPPS 

CCAS 

(d) 

User-friendly, 

menu-driven  screens 

X 

X 

(j) 

Variance  analysis 

X 

X 

X 

X 

with  indicators  of  major 

problem  areas 

X 

X 

(i) 

Regression  analysis 

X 

(e) 

Installation 

X 

X 

X 

X 

(f) 

Productivity 

X 

X 

(s) 

Estimate  at  Completion  (EAC): 

Several  different  EAC 

calculations 

X 

X 

X 

X 

Guide  to  determine  which 

EAC  to  use 

(g) 

Ability  to  print  on  plotter 

or  printers 

X 

X 

X 

X 

(h) 

Ability  to  print  graphics 

or  data.* 

Data  on  printer 

X 

X 

X 

X 

Graphics  on  printer 

Graphics  on  plotter 

X 

X 

(unable  to  evaluate) 

X 

X 

(n) 

Ability  to  modify  formulas 

and  update  charts: 

Update  data  *** 

X 

X 

X 

X 

Modify  formulas 

Update  graphs  *** 

X 

X 

X 

(q) 

Software  is  available  on 

personal  computer 

X 

X 

X 

X 

(r) 

Interactive  MIS  with 

contractor: 

Data  received  electronically 

from  contractor  **** 

X 

X 

Data  received  on  disk 

from  contractor 

X 

X 

X 

(1) 

Narrative  section 

X 

X 

(m) 

Documentation 

X 

X 

X 

(a) 

Help  indexes 

X 

X 

***  It  is  difficult  to  update  data  on  CAPPS  and  CCAS  and  difficult  to 
update  graphs  on  PA  and  CCAS . 

****  CCAS  provides  an  interactive  MIS  with  the  contractor  only  for  DAES 
programs . 
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IV.  Results.  Conclusions  and  Recomendations 


Selection  of  ’’the  Ideal"  CPR  Software  Package  or  Features  of  "the  Ideal" 
Software  Package 

As  stated  previously.  Expert  Choice  is  a  decision  support  software 
program  and  was  used  in  ranking  the  four  CPR  analysis  software  packages 
against  the  fourteen  "most  important"  user-established  criteria.  Expert 
Choice  allows  the  user  to  evaluate  criteria  in  either  a  verbal  or 
numerical  mode.  Therefore,  models  were  developed  and  compared  for  both 
types  of  modes.  Appendix  B  contains  the  detailed  analysis  for  the  verbal 
and  numerical  modes  of  selecting  "the  ideal”  CPR  analysis  software 
package  using  Expert  Choice.  A  sensitivity  analysis  was  also  performed 
on  both  the  verbal  and  numerical  modes  with  regard  to  the  importance  of 
the  evaluation  criteria  in  yanking  the  software  packages. 

Expert  Choice  Verbal  Mode.  The  first  step  in  using  Expert  Choice  to 
evaluate  the  CPR  analysis  software  packages  against  the  user-established 
criteria  was  to  evaluate  the  criteria  against  each  other  with  regard  to 
order  of  importance.  Then  the  software  packages  were  evaluated  against 
each  other  with  respect  to  each  criterion.  The  verbal  scale  used  to 
compare  the  criteria  are  as  follows:  Equal,  Moderate,  Strong,  Very 
Strong  and  Extreme.  In  addition  to  this,  the  user  may  select  a 
comparison  alternative  that  is  in-between  the  options  listed  above.  Por 
example,  "Equal  to  Moderately  More  Important"  or  "Very  Strong  to 
Extremely  More  Important"  alternatives  may  be  chosen. 

The  comparison  was  conducted  in  three  basic  steps.  The  criteria 
evaluated  against  each  other  first  were  the  seven  categories  listed  in 
Table  8:  FRIENDLY,  TOOLS,  MOD/PRNT,  PC,  INTERACT,  DOCUMENT  and  HELP.  As 
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can  be  seen  in  Table  8,  the  FRIENDLY  and  MOD/PRNT  categories  consist  of 
three  criteria  and  the  TOOLS  category  consists  of  four  criteria.  In 
determining  the  overall  weight  of  the  FRIENDLY,  TOOLS  and  MOD/PRNT 
categories,  the  weights  of  the  respective  subcriteria  in  each  category 
were  averaged.  The  results  are  as  follows:  FRIENDLY  =  18.333,  TOOLS  = 
16.24975  and  MOD/PRNT  =  12.888667.  The  second  basic  step  consisted  of 
evaluating  the  subcriteria  in  each  of  the  three  general  categories 
(FRIENDLY,  TOOLS  and  MOD/PRNT)  against  each  other.  Finally,  the  third 
step  consisted  of  evaluating  the  four  CPR  analysis  software  packages 
against  each  other  with  respect  to  each  of  the  fourteen  criteria.  The 
details  of  the  analysis  are  contained  in  Appendix  B.  The  results  of  the 
verbal  mode  comparison  method  are  listed  in  Table  11. 

TABLE  11 

RANKINGS  OF  FOUR  CPR  ANALYSIS  SOFTWARE  PROGRAMS 
USING  THE  EXPERT  CHOICE  VERBAL  MODE 


CPR  Analysis 

Software  Program  Ranking 

PA  0.479 

CAPPS  0.209 

CCAS  0.176 

CPR-EZ  0.136 

1.000 


Sensitivity  Analysis  of  Verbal  Mode.  A  sensitivity  analysis 
was  performed  with  regard  to  the  importance  of  the  evaluation  criteria  in 
ranking  the  software  packages.  That  is,  the  sensitivity  analysis 
calculated  how  sensitive  the  final  decision  would  be  to  a  change  in  the 
weightings  of  the  fourteen  criteria.  When  comparing  each  of  the  seven 
general  categories  (FRIENDLY,  TOOLS,  MOD/PRNT,  PC,  INTERACT,  DOCUMENT  and 
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HELP)  to  the  goal  of  selecting  the  ideal  software  package,  Performance 
Analyzer  (PA)  was  selected  for  each  category  with  one  exception.  The 
HELP  criterion  received  a  weighting  in  the  verbal  mode  of  0.023  with 
respect  to  the  other  six  general  categories  (see  Appendix  B) .  During  the 
sensitivity  analysis,  Performance  Analyzer  was  chosen  as  the  ideal 
software  package  with  respect  to  the  HELP  criterion  weighting  of  0.023. 
However,  if  the  weight  for  the  HELP  criterion  was  approximately  0.45  or 
greater,  the  CAPPS  software  program  would  have  been  selected.  In 
addition  to  this,  if  the  M0D/PRNT  criterion  had  a  weighting  of  1.00 
instead  of  0.119,  the  PA  and  CCAS  software  packages  would  both  have  been 
selected  as  the  ideal  CPR  analysis  software  package.  Also,  if  the 
weighting  for  the  PC  criterion  was  1.00  instead  of  0.094,  all  four  of  the 
software  packages  would  have  been  equally  selected. 

Sensitivity  analysis  was  also  performed  on  the  three  general 
categories  (FRIENDLY,  TOOLS  and  M0D/PRNT)  which  contained  the  following 
subcriteria : 


PRIENDLY: 

Criterion  (d):  User  friendly,  menu-driven  screens  (SCREENS) 
Criterion  (e):  Installation  (INSTALL) 

Criterion  (f):  Productivity  (PR0DTVTY) 

TOOLS: 


Criterion  (j): 
Criterion  (i): 
Criterion  (s): 
Criterion  (1): 


Variance  analysis  (VARIANCE) 
Regression  analysis  (REGRESS) 
Estimate  at  Completion  (EAC) 
Narrative  section  (NARRATIV) 
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MOD/PRNT: 

Criterion  (g):  Ability  to  print  on  plotter  or  printers  (PRTPLPRN) 
Criterion  (h):  Ability  to  print  graphics  or  data  (PRTGRDTA) 

Criterion  (n):  Ability  to  modify  formulas  and  update  charts  (MODIPY) 
The  results  from  performing  sensitivity  analysis  on  the  subcriteria 
in  the  FRIENDLY  category  are  that  the  Performance  Analyzer  (PA)  software 
package  was  chosen  for  all  weightings  of  the  PRODTVTY  criterion  and  for 
all  weightings  of  approximately  0.05  and  above  for  the  SCREENS  criterion. 
PA  was  also  chosen  as  the  ideal  software  package  with  respect  to  the 
INSTALL  criterion  weighting  of  0.258.  However,  if  the  weight  for  the 
INSTALL  criterion  was  approximately  0.65  or  greater,  the  CPR-EZ  software 
program  would  have  been  selected. 

The  results  from  performing  sensitivity  analysis  on  the  subcriteria 
in  the  TOOLS  category  are  that  the  Performance  Analyzer  (PA)  software 
package  was  chosen  for  all  weightings  with  regard  to  each  of  the  four 
subcriteria  (VARIANCE,  REGRESS,  EAC  and  NARRATIV) . 

The  results  from  performing  sensitivity  analysis  on  the  subcriteria 
in  the  MOD/PRNT  category  are  that  Performance  Analyzer  was  chosen  for  the 
MODIFY  subcriteria  at  its  weighting  of  0.124.  However,  the  CCAS  software 
program  was  selected  for  the  PRTPLPRN  and  PRTGRDTA  subcriteria  at  the 
weightings  assigned  to  those  criteria  (0.517  for  PRTPLPRN  and  0.359  for 
PRTGRDTA) .  Further  analysis  revealed  that  the  CCAS  software  program 
would  be  selected  as  the  ideal  software  package  for  a  weighting  of 
approximately  0.120  or  less  for  the  MODIFY  subcriterion.  However, 
Performance  Analyzer  (PA)  would  be  selected  with  a  weighting  of  approx- 
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imately  0.34  or  less  for  the  PRTGRDTA  criterion  and  a  weighting  of 
approximately  0.40  or  less  for  the  PRTPLPRN  criterion. 

Expert  Choice  Numerical  Mode.  As  stated  previously,  the  first  step 
in  using  Expert  Choice  to  evaluate  the  CPR  analysis  software  packages  was 
to  evaluate  the  criteria  against  each  other  with  regard  to  order  of 
importance.  The  same  approach  was  taken  as  in  the  verbal  mode.  That  is, 
first  the  seven  general  categories  (FRIENDLY,  TOOLS,  MOD/PRNT,  PC, 
INTERACT,  DOCUMENT  and  HELP)  listed  in  Table  8  were  compared.  Then  the 
subcriteria  in  the  PRIENDLY,  TOOLS  and  MOD/PRNT  categories  were  compared. 
Finally,  the  four  CPR  analysis  software  packages  were  evaluated  against 
each  other  with  respect  to  each  of  the  fourteen  criteria. 

The  seven  general  categories  and  the  subcriteria  in  the  FRIENDLY, 
TOOLS  and  MOD/PRNT  categories  were  evaluated  based  on  the  weights  in 
Table  8  for  a  numerical  analysis.  The  overall  weights  of  the  FRIENDLY, 
TOOLS  and  MOD/PRNT  categories  were  determined  by  averaging  the  weights  o'f 
the  respective  subcriteria  in  each  category.  The  weights  used  in 
evaluating  the  seven  general  categories  as  well  as  the  subcriteria  in 
three  of  those  categories  (FRIENDLY,  TOOLS  and  MOD/PRNT)  are  listed  below 
in  Table  12.  The  CPR  analysis  software  packages  were  evaluated  using 
the  same  comparisons  conducted  in  the  verbal  analysis  mode. 
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TABLE  12 


WEIGHTS  FOR  THE  GENERAL  CRITERIA  AND  SUBCRITERIA  CATEGORIES 
IN  CONDUCTING  A  NUMERICAL  ANALYSIS  OF  THE  CPR  SOFTWARE  PACKAGES 


Seven  General 

Subcriteria 

Weight 

Criteria  Categories 

Weight 

Categories 

(Same  as  Table 

FRIENDLY 

18.333 

SCREENS 

21.0 

INSTALL 

18.333 

PRODTVTY 

15.667 

TOOLS 

16.24975 

VARIANCE 

19.333 

REGRESS 

19.333 

EAC 

15.333 

NARRATIV 

11.0 

MOD/PRNT 

12.888667 

PRTPLPRN 

13.333 

PRTGRDTA 

13.0 

MODIFY 

12.333 

PC 

12.333 

• 

INTERACT 

11.333 

DOCUMENT 

9.333 

- 

HELP  8.667 


The  results  of  the  numerical  mode  comparison  method  are  listed  in 
Table  13.  The  details  of  the  analysis  are  contained  in  Appendix  B. 


TABLB  13 

RANKINGS  OF  FOUR  CPR  ANALYSIS  SOFTWARE  PROCRAMS 
USING  THE  EXPERT  CHOICE  NUMERICAL  MODB 


CPR  Analysis 

Software  Program  Ranking 

PA  0.460 

CAPPS  0.213 

CCAS  0.189 

CPR-EZ  0.138 

1.000 
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Sensitivity  Analysis  of  Numerical  Mode.  A  sensitivity  analysis 


was  performed  on  the  numerical  mode  of  Expert  Choice  as  well  as  on  the 
verbal  mode.  When  comparing  each  of  the  seven  general  categories 
(FRIENDLY,  TOOLS,  MOD/PRNT,  PC,  INTERACT,  DOCUMENT  and  HELP)  to  the  GOAL 
of  selecting  the  ideal  CPR  analysis  software  package,  Performance 
Analyzer  (PA)  was  selected  for  each  category  with  one  exception.  The 
HELP  criterion  received  a  weighting  in  the  numerical  mode  of  0.09?  with 
respect  to  the  other  six  general  categories  (see  Appendix  B) .  During  the 
sensitivity  analysis.  Performance  Analyzer  was  chosen  as  the  ideal 
software  package  with  respect  to  the  HELP  criterion  weighting  of  0.097. 
However,  if  the  weight  for  HELP  was  approximately  0.54  or  greater,  the 
CAPPS  software  program  would  have  been  selected.  In  addition  to  this,  if 
the  weighting  for  the  PC  criterion  was  1.00  instead  of  0.139,  all  four  of 
the  software  packages  would  have  been  equally  selected. 

Sensitivity  analysis  was  then  performed  on  the  subcriteria  in  each 
of  the  general  categories  of  PRIENDLY,  TOOLS  and  MOD/PRNT.  The  results 
from  performing  sensitivity  analysis  on  the  subcriteria  in  the  FRIENDLY 
category  are  that  the  Performance  Analyzer  (PA)  software  package  was 
chosen  for  all  weightings  of  the  PR0DTVTY  and  SCREENS  criteria.  PA  was 
also  chosen  as  the  ideal  software  package  with  regard  to  the  INSTALL 
criterion  weighting  of  0.341.  However,  if  the  weight  for  the  INSTALL 
criterion  was  approximately  G.?0  or  more,  the  CPR-EZ  software  program 
would  have  been  selected. 

The  results  from  performing  sensitivity  analysis  on  the  subcriteria 
in  the  TOOLS  category  are  that  the  Performance  Analyzer  (PA)  software 
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package  was  chosen  for  all  weightings  with  regard  to  each  of  the  four 
subciiteria  (VARIANCE,  REGRESS,  EAC  and  NARRATIV) . 

The  results  from  performing  sensitivity  analysis  on  the  subcriteria 
in  the  MOD/PRNT  category  are  that  Performance  Analyzer  was  chosen  for  the 
MODIFY,  PRTGRDTA  and  PRTPLPRN  subcriteria  at  the  weightings  assigned  to 
those  subcriteria  (0.313  for  MODIFY  and  0.344  for  both  PRTGRDTA  and 
PRTPLPRN).  Further  analysis  revealed  that  the  CCAS  software  program 
would  be  selected  as  the  ideal  software  package  at  a  weighting  of  approx¬ 
imately  0.13  or  less  for  the  MODIFY  subcriterion,  approximately  0.62  or 
more  for  the  PRTGRDTA  criterion  and  approximately  0.71  or  more  for  the 
PRTPLPRN  criterion. 

Conclusions  and  Reconmendations  from  Comparing  the  Results  of  the 
Verbal  and  Numerical  Modes  of  Expert  Choice.  In  comparing  the  results 
from  the  verbal  and  numerical  analysis  of  the  CPR  software  packages  using 
Expert  Choice  as  well  as  the  sensitivity  analysis  that  was  performed. 
Performance  Analyzer  (PA)  is  selected  as  "the  most  ideal"  CPR  analysis 
software  package  from  the  four  packages  that  were  evaluated.  However, 
Performance  Analyzer  does  not  completely  fulfill  the  requirements  for 
"the  Ideal"  CPR  analysis  software  package  established  by  the  AFSC  product 
divisions.  For  example,  PA  does  not  include  a  guide  to  determine  which 
Estimate  at  Completion  (EAC)  calculation  to  use  for  a  given  program 
and/or  situation.  Comments  on  each  CPR  analysis  software  package  with 
respect  to  the  user-established  criteria  are  available  in  Phase  V  of 
Chapter  III. 
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Further  Study 


In  conducting  research  for  this  study,  a  question  was  raised  by 
Lt  Col  Thornes  Bowman  of  ASD/ACCM  (AV  785-6101)  regarding  whether  the 
"real"  information  needs  of  the  user  with  respect  to  C/SCSC  are  being 
fulfilled.  The  answer  to  this  question  may  require  a  revised  format  for 
the  CPR  and  CSSR.  Because  a  survey  of  this  nature  was  beyond  the  scope 
of  this  thesis,  it  was  not  conducted  as  a  part  of  this  project.  However, 
the  following  questions  may  assist  in  determining  whether  or  not  the 
C/SCSC  information  needs  of  the  program  manager  and/or  cost  analyst  are 
currently  being  fulfilled: 

1.  What  System  Program  Office  (SPO)  do  you  work  in  and  how  many  years 
have  you  been  assigned  to  this  SPO? 

2.  As  a  manager  on  this  program,  what  are  you  looking  for  in  C/SCSC 
data — what  are  your  real  data  needs  to  help  make  decisions? 

3.  Does  current  CPR/CSSR  information  contain  all  of  the  information  you 
need  to  run  the  program  adequately  (if  not,  what  is  missing)? 

4.  If  CPR  data  are  not  appropriate  for  subprogram  management 

(ex:  avionics)  or  high  risk  control,  what  additional  information 
would  assist  you  in  this  process? 

5.  For  your  purposes,  does  CPR/CSSR  information  need  to  be  automated — 
why  or  why  not  (and  if  yes ,  what  features  does  the  software  need  to 
include)? 

A  Trainee  Research  Project  titled  "ASD  Standards  for  Program 
Analysis  of  CPR/CSSR"  by  Charlotte  Mathena,  dated  January  10,  1990,  is 
available  in  the  ASD  Cost  Library.  During  this  research  project, 

CPR/CSSR  experts  from  Aeronautical  Systems  Division  (ASD),  Electronic 
Systems  Division  (ESD),  Space  Systems  Division  (SSD),  Missile  Systems 
Division  (MSD),  Air  Force  Institute  of  Technology  (AFIT),  Defense  Systems 
Management  College  (DSMC)  and  Air  Force  Systems  Conmand  (AFSC)  were  inter¬ 
viewed  with  regard  to  determining  what  information  needs  to  be  included 
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in  conducting  an  effective  CPR  and  CSSR  analysis.  The  information 
obtained  from  this  study  may  assist  in  determining  the  "real"  C/SCSC 
information  needs  of  the  user. 
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Appendix  A;  CPR  Analysis  Software  Packages 


The  CPR  analysis  software  packages  that  were  analyzed  include  the 
following: 

Space  Systems  Division  (SSD)  Performance  Analyzer  (PA) 

Air  Force  Cost  Center  CPR-EZ 

Air  Force  Cost  Center  Contractor  Cost  Analysis  System  (CCAS) 

Defense  Systems  Management  College  (DSMC)  Contract  Appraisal 

System  (CAPPS) 


Space  Systems  Division  (SSD)  Performance  Analyzer  (PA): 

a.  SSD's  performance  analyzer  (PA)  was  developed  [by  Thomas /Seif ers , 
Inc.  with  the  assistance  of  Headquarters  Space  Division  (SD)]  to 
streamline  and  automate  the  analysis  and  reporting  processes 
associated  with  analyzing  the  SSD  contractor's  CPRs  and  C/SSR. 

b.  The  PA  features  intuitive  metiuing,  a  contract  probe  module, 
visual  displays  of  the  WBS  and  functional  structure,  and  graphic 
displays  of  the  performance  data.  It  is  designed  to  serve  the  needs 
of  contract  financial  analysts,  engineers,  and  program  managers 
alike. 

(1)  The  PA  is  extremely  user-friendly,  allowing  the  user  to 
visually  define  the  WBS  and  functional  tree  structure  and  enter  and 
edit  data. 

(2)  The  analysis  module  has  been  designed  to  be  extremely  easy 
to  use.  The  user  may  probe  down  the  tree  structures  using  on-screen 
performance  indicators;  switch  from  WBS  to  functional  structures, 
viewcharts,  and  reports  from  each  element;  or  jump  from  one  element 
to  another. 

(3)  All  graphs  and  reports  can  be  displayed  on  the  screen  and 
printed  or  plotted  to  selected  hard-copy  devices. 

c.  PA  displays,  prints,  and  plots  graphs  such  as  analysis-level 
trend  data  at  the  lowest  levels  and  quality  briefing  charts  at  the 
highest  levels  for  program  management  reviews.  It  also  produces 
hard-copy  reports  for  internal  program  office  use.  It  fully 
supports  the  automated  transfer  of  data  from  the  contractor  to  the 
program  office,  to  the  headquarters  staffs,  to  the  Executive 
Mar.-gement  System  (EIS) ,  and  then  to  OSD  through  the  Defense 
Acquisition  Executive  Summary  [DABS]  data  base. 

d.  By  placing  a  standardized  CDRL  [Contract  Data  Requirements  List] 
on  the  contracts,  the  contractor  will  submit  the  CPR  and/or  C/SSR  on 
a  network  system  or  a  floppy  dink  in  a  structured  ASCII  format  to 
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the  program  office.  It  requires  little  to  no  investment  by  the 
contractors.  The  program  offices  will  then  import  the  data  into 
their  PA  systems  and  do  their  analysis.  Each  analyst  can  input 
their  program  office  EACs  for  each  lower-level  WBS  and/or  functional 
element  along  with  narrative  text  explanations  of  the  EAC 
methodology  they  used. 

e.  PA  will  make  the  following  computations: 

(1)  Mathematical  checks  for  data  errors. 

(2)  Percent  complete  by  WBS/f unction. 

(3)  Current  and  cumulative  CV  and  SV. 

(4)  CPIs  and  SPIs  for  current  and  cumulative  data. 

(5)  VAC. 

(6)  TCPI . 

(7)  SV  verification  in  weeks  and  days. 

(8)  Tolerance  bands. 

(9)  Three-month  moving  average  calculations. 

(10)  EACs  using  weighted  indices. 

f.  PA  [Version  2.6]  supports  the  IBM  PC/XT/AT  compatible  computers 
(Z-150s,  Z-248s,  etc.);  EGA,  Hercules,  VGA  graphic  cards;  all  Epson 
compatible  printers,  most  laser- jet  printers;  DOS  (version  2.0  or 
higher);  and  the  Hewlett-Packard  7550A  plotter. 

g.  The  PA  software  package  consists  of  three  floppy  diskettes  (they 
are  not  copy-protected)  and  a  user's  manual.  It  must  reside  on  your 
computer's  hard  drive  as  it  will  not  run  on  the  floppies.  The 
config.sys  file  on  the  root  directory  of  your  boot  disk  must 
contain:  buffers=20  and  files=20.  Without  these  comnands,  the 
system  will  not  work  properly. 

h.  To  receive  copies  of  the  PA,  please  write  or  call  SSD/ACCI,  Los 

Angeles  APB,  PO  Box  92960,  Los  Angeles,  CA  90009-2960,  (213)643-1520 
or  AUTOVON  833-1520.  (3:33-34) 


Air  Porce  Cost  Center  CPR-EZ: 

a.  This  model  was  developed  by  the  Air  Force  Cost  Center  to  provide 
a  general  tool  to  perform  detailed  CPR  analysis.  It  allows  the 
analyst  to  work  at  any  level  of  the  work  breakdown. 

b.  The  program  wil  make  the  following  calculations: 

(1)  CV. 

(2)  SV. 

(3)  CPI. 

(4)  SPI. 

(5)  PMB. 

(6)  Percent  complete. 

(7)  Pive  EAC  calculations  with  flexibility. 
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c.  Hardware  and  software  requirements  are  as  follows: 

(1)  IBM-compatible,  dual  floppy  or  Winchester  hard  drive. 

(2)  Graphics  board  within  the  PC,  color  monitor  not  required. 

(3)  Mo  specific  printer  requirements. 

(4)  MS-DOS. 

(5)  LOTUS  1-2-3  version  2.01. 

d.  Contact  Department  of  the  Air  Force,  Attn:  SAP /ACC,  Pentagon, 
Room  4D159,  Washington,  D.C.  20330-1000,  AUT0V0N  225-4510  for  more 
information.  (3:34-35) 


Air  Force  Cost  Center  Contractor  Cost  Analysis  System  (CCAS): 

a.  This  program  allows  the  analyst  to  do  cost  performance  analysis 
directly  from  the  DABS  disk  using  an  IBM-compatible  PC.  It  will 
create  and  maintain  an  historical  data  base  using  information  from 
several  of  the  pertinent  DABS  formats.  CCAS  will  also  accept 
keyboard  entry  of  non-DAES  data. 

b.  The  program  will  make  the  following  current  and  cumulative 


calculations : 

(1) 

CV. 

(2) 

sv. 

(3) 

SPI. 

(4) 

CPI. 

(5) 

TCPI . 

(6) 

Percent  complete. 

(7) 

Four  EAC  calculations  with  flexibility 

r,  Hardware  and  software  requirements  are  as  follows: 

(1)  IBM  time-compatible  dual  floppy  or  Winchester  hard  drive. 

(2)  Graphics  board  within  the  PC,  color  monitor  not  required. 

(3)  Epson  or  IBM-compatible  printer. 

(4)  HP  7475A  plotter  or  compatible. 

(5)  MS-DOS  Version  2.0  or  later. 

d.  Contact  Department  of  the  Air  Force,  Attn:  SAF/ACC,  Pentagon, 
Room  4D159,  Washington,  D.C.  20330-1000,  AUT0V0N  225-4640  for  more 
information.  (3:35) 


DSMC  Contract  Appraisal  System  (CAPPS): 

a.  The  Defense  Systems  Management  College  (DSMC)  CAPPS  [was  devel¬ 
oped  by  DAI,  Inc.  in  Rockville,  Maryland  and]  is  a  management  tool 
to  aid  executives  and  program  managers  in  their  understanding  of 
cost  and  schedule  status  of  their  program.  This  system  is  designed 
to  support  the  executive  or  program  manager  with  high-level , 
summarized  cost  and  schedule  status  information.  CAPPS  will  make 
the  following  computations: 
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(1) 

CV. 

(2) 

sv. 

(3) 

VAC. 

(4) 

CV  percent. 

(5) 

CPI. 

(6) 

Index  to  go. 

(7) 

EAC  techniques. 

(8) 

EAC  formulas. 

b.  Some  of  the  above  computations  can  be  displayed  graphically. 
Hardware  requirements  to  run  CAPPS  are  an  IBM  PC/XT  or  a  true 
compatible  or  a  Zenith  110/120;  one  floppy  disk  drive;  192KB  RAM 
minimum;  PCDOS  2.X  operating  system  for  the  IBM  or  an  MSDOS  2.X  for 
the  Zenith.  Further  information  about  the  use  of  this  system  can  be 
obtained  from  PMSS  Directorate  (DRI-S),  DSMC,  (703)664-4795/5783  or 
AUTOVON  354-4795/5783.  (3:36-37) 


Automated  Financial  Analysis  (APA)  Tabular  Report  Generator: 

NAVAIR  is  currently  updating  Electronic  Systems  Division's  (ESD) 

Automated  Financial  Analysis  (APA)  program  so  that  it  will  be  available 

in  a  PC  version.  The  point  of  contact  for  the  Navy  is  AIR-713  (AFA 

Project  Team)  at  (202)  433-2653/5038  or  AV  288-2653/5038. 

Following  is  a  description  of  the  ESD  Automated  Financial  Analysis 

(AFA)  Program  that  was  included  in  a  draft  version  (dated  1  October  1988) 

but  excluded  from  the  final  version  of  AFSCP  173-4,  Cost  Analysis:  Guide 

to  Analysis  of  Contractor  Cost  Data,  dated  1  September  1989: 

a.  The  ESD  AFA  model  was  desgined  to  meet  the  needs  of  the 
financial  manager  in  analysis  of  both  the  CPR  and  the  C/SSR.  In 
addition  to  calculating  a  cost-at-completion  discussed  below,  this 
model  also  contains  several  other  routines: 

b.  Pirst  of  all,  after  the  data  from  the  cost  report  is  entered, 
this  data  is  displayed  in  two  segments  -  one  for  current  month  data; 
one  for  cumulative  to  date  data.  The  mathematics  of  these  two 
segments  are  then  checked,  and  errors  are  pinpointed  and  flagged  in 
an  error  listing  table. 

c.  The  schedule  variance  feature  gives  an  assessment  of  the  comple¬ 
tion  status  of  the  contract.  Schedule  Variance  Percentages  (SVP) 
are  computed  using  the  formula 
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x  100 


An  SVP  greater  than  zero  reflects  an  ahead-of  schedule  situation; 
whereas,  a  SVP  of  less  than  zero  indicates  a  behind  schedule  status. 
Months  behind  schedule  is  also  computed  by  calculating  the 
historical  point  in  time  when  the  currently  reported  cum  BCWP  equals 
previously  reported  cum  BCWS. 


d.  The  Cost  Variance  Percent  (CVP)  is  calculated  using  the  formula 


BCWP  ~  ACWP  x  100 

BCWP 


A  CVP  greater  than  zero  reflects  an  under-cost  situation;  whereas  a 
CVP  less  than  zero  reflects  an  over-cost  position. 

e.  The  completion  status  is  determined  using  a  series  of  formulas. 
It  displays  what  percentage  should  be  done,  what  percentage  actually 
is  done,  and  what  percent  of  the  funds  were  spent  to  accomplish  the 
task. 


f.  G&A  rates  are  also  analyzed  by  the  BSD  AFA  model.  It  displays 
the  historical  rates,  and  calculates  the  projected  rates  and  rates 
at  completion. 

g.  A  potential  price  at  completion  is  determined  for  each  cost  at 
completion  method  discussed  below,  for  the  BAC  and  for  the  LRE. 

This  includes  the  Point  of  Total  Assumption  (PTA)  calculation  based 
upon  the  contract  type  and  share  ratios . 

h.  Cost  At  Completion  (CAC)  can  be  calculated  with  six  different 
formulae  all  of  which  utilize  a  Performance  Factor  (PF).  Selection 
of  the  most  appropriate  formula  is  left  to  the  discretion  of  the 
analyst.  Each  CAC  methodology  is  discussed  below: 

(1)  Method  1  -  This  method  is  calculated  using  a  performance 
factor  based  on  current  month  cost  performance. 

(2)  Method  2  -  This  method  is  calculated  using  a  performance 
factor  based  upon  a  three  (3)  month  moving  average  of  current  month 
cost  performance. 

(3)  Method  3  -  This  method  is  calculated  using  a  performance 
factor  based  on  cumulative  cost  performance. 

(4)  Method  4  -  This  method  is  calculated  using  a  performance 
factor  provided  by  program  office  personnel. 

(5)  Method  5  -  This  method  is  calculated  using  a  performance 
factor  based  on  the  last  quarter  cost  performance. 
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(6)  Method  6  -  This  method  is  calculated  using  a  performance 
factor  based  on  a  combination  of  cost  and  schedule  variance  and  user 
chosen  performance  weightings. 

i.  The  source  deck  for  the  AFA  is  available  thiough  BSD.  It  is 
compatible  with  CDC  6600  computer  hardware  and  associated  fortran 
compiler.  In  addition,  other  DOD  agencies  have  made  the  program 
compatible  with  the  GB  time  sharing  system.  To  obtain  the  source 
deck  and  to  gather  further  information  on  how  to  access  the  program, 
contact  ESD/ACCI,  (617)  377-5223  or  AUT0V0N  478-5223. 
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Appendix  B:  Detailed  Analysis  of  Selecting 
"the  Ideal"  CPR  Analysis  Software  Package  Using  Expert  Choice 


Expert  Choice  Verbal  Mode 

The  verbal  scale  used  to  compare  the  fourteen  "most  important" 
user-established  criteria  and  the  four  CPR  analysis  software  packages 
is  listed  below: 

Extreme  - 

Very  Strong  - 

Strong  - 

Moderate  - 

Equal  - 


The  comparison  was  conducted  in  three  basic  steps.  The  criteria 
evaluated  against  each  other  first  were  the  following  seven  general 
categories:  FRIENDLY,  TOOLS,  MOD/PRNT,  PC,  INTERACT,  DOCUMENT  and  HELP. 


FIRST  STEP  (With  Respect  to  GOAL): 

FRIENDLY  is  Moderately  More  Important  than  TOOLS 

FRIENDLY  is  Moderate  to  Strongly  More  Important  than  MOD/PRNT 

FRIENDLY  is  Strong  to  Very  Strongly  More  Important  than  INTERACT 

FRIENDLY  is  Strongly  More  Important  than  PC 

FRIENDLY  is  Very  Strong  to  Extremely  More  Important  than  HELP 

FRIENDLY  is  Very  Strongly  More  Important  than  DOCUMENT 

TOOLS  is  Moderate  to  Strongly  More  Important  than  MOD/PRNT 
TOOLS  is  Strong  to  Very  Strongly  More  Important  than  INTERACT 
TOOLS  is  Stroiigiy  More  Important  than  PC 

TOOLS  is  Very  Strong  to  Extremely  More  Important  than  HELP 
TOOLS  is  Very  Strongly  More  Important  than  DOCUMENT 

MOD/PRNT  is  Moderately  More  Important  than  INTERACT 
MOD/PRNT  is  Equal  to  Moderately  More  Important  than  PC 
MOD/PRNT  is  Strongly  More  Important  than  HELP 
MOD/PRNT  is  Moderate  to  Strongly  More  Important  than  DOCUMENT 

PC  is  Moderately  More  Important  than  INTERACT 

INTERACT  is  Moderate  to  Strongly  More  Important  than  HELP 
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INTERACT  is  Moderately  More  Important  than  DOCUMENT 

PC  is  Strongly  More  Important  than  HELP 

PC  is  Moderate  to  Strongly  More  Important  than  DOCUMENT 

DOCUMENT  is  Moderately  More  Important  than  HELP 


Inconsistency  Ratio  =  0.079 

(An  Inconsistency  Ratio  of  0.1  or  more  indicates  inconsistent  judgments.) 


RESULTS : 


FRIENDLY 

0.389 

TOOLS 

0.281 

MOD / PRNT 

0.119 

INTERACT 

0.058 

PC 

0.094 

HELP 

0.023 

DOCUMENT 

0.036 

1.000 


The  second  basic  step  consisted  of  evaluating  the  subcriteria  in 
each  of  the  following  three  general  categories  against  each  other: 
FRIENDLY,  TOOLS  and  M0D/PRNT. 

SECOND  STEP: 

With  Respect  to  FRIENDLY  (0.389): 

SCREENS  is  Moderately  More  Important  than  INSTALL 
SCREENS  is  Strongly  More  Important  than  PR0DTVTY 

INSTALL  is  Moderately  More  Important  than  PR0DTVTY 


Inconsistency  Ratio  =  0.033 


RESULTS : 


SCREENS  0.637 
INSTALL  0.258 
PR0DTVTY  0.105 

1.000 
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With  Respect  to  TOOLS  (0.281): 

VARIANCE  is  Equally  as  Important  as  REGRESS 

VARIANCE  is  Moderate  to  Strongly  More  Important  than  EAC 

VARIANCE  is  Strong  to  Very  Strongly  More  Important  than  NARRATIV 

REGRESS  is  Moderate  to  Strongly  More  Important  than  EAC 
REGRESS  is  Strong  to  Very  Strongly  More  Important  than  NARRATIV 

EAC  is  Moderate  to  Strongly  More  Important  than  NARRATIV 


Inconsistency  Ratio  =  0.045 


RESULTS : 


VARIANCE 

0.406 

REGRESS 

0.406 

EAC 

0.134 

NARRATIV 

0.054 

1.000 

With  Respect  to  MOD/PRNT  (0.119): 

PRTGRDTA  is  Moderate  to  Strongly  More  Important  than  MODIFY 

PRTPLPRN  is  Moderately.  More  Important  than  MODIFY 
PRTPLPRN  is  Equal  to  Moderately  More  Important  than  PRTGRDTA 

Inconsistency  Ratio  =  0.093 


results: 


PRTPLPRN  0.517 
PRTGRDTA  0.359 
MODIFY  0.124 

1.000 


The  third  step  consisted  of  evaluating  the  four  CPR  analysis 
software  packages  against  each  other  with  respect  to  each  of  the  fourteen 
criteria. 
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THIRD  STEP: 


FRIENDLY  Category  (0.389): 

With  Respect  to  SCREENS  (0.637): 

PA  is  Extremely  More  Preferable  than  CPR-EZ 

PA  is  Moderate  to  Strongly  More  Preferable  than  CAPPS 

PA  is  Very  Strongly  More  Preferable  than  CCAS 

CCAS  is  Moderate  to  Strongly  More  Preferable  than  CPR-EZ 

CAPPS  is  Very  Strongly  More  Preferable  than  CPR-EZ 
CAPPS  is  Strongly  More  Preferable  than  CCAS 


Inconsistency  Ratio  =  0.113 


RESULTS*. 


PA 

CAPPS 

CCAS 

CPR-EZ 


0.610 

0.262 

0.089 

0.039 

1.000 


With  Respect  to  INSTALL  (0.258): 

CPR-EZ  is  Moderately  More  Preferable  than  PA 

CCAS  is  Moderately  More  Preferable  than  PA 

PA  is  Equal  to  Moderately  More  Preferable  than  CAPPS 

CPR-EZ  is  Equal  to  Moderately  More  Preferable  than  CCAS 
CPR-EZ  is  Moderately  More  Preferable  than  CAPPS 

CCAS  is  Moderately  More  Preferable  than  CAPPS 


Inconsistency  Ratio  =  0.045 


RESULTS : 


PA 

CPR-EZ 

CCAS 

CAPPS 


0.146 

0.439 

0.311 

0.104 

1.000 
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With  Respect  to  PRODTVTY  (0.105): 

PA  is  Extremely  More  Preferable  than  CPR-EZ 

PA  is  Very  Strongly  More  Preferable  than  CCAS 

PA  is  Moderate  to  Strongly  More  Preferable  than  CAPPS 

CCAS  is  Moderately  More  Preferable  than  CPR-EZ 

CAPPS  is  Very  Strong  to  Extremely  More  Preferable  than  CPR-EZ 
CAPPS  is  Strong  to  Very  Strongly  More  Preferable  than  CCAS 


Inconsistency  Ratio  =  0.107 


RESULTS : 


PA 

0.607 

CPR-EZ 

0.039 

CCAS 

0.076 

CAPPS 

0.279 

1.000 

TOOLS  Category  (0.281): 

With  Respect  to  VARIANCE  (0.406): 

PA  is  Extremely  More  Preferable  than  CPR-EZ 

PA  is  Very  Strongly  More  Preferable  than  CCAS 

PA  is  Equal  to  Moderately  More  Preferable  than  CAPPS 

CCAS  is  Moderatley  More  Preferable  than  CPR-EZ 

CAPPS  is  Very  Strong  to  Extremely  More  Preferable  than  CPR-EZ 
CAPPS  is  Strong  to  Very  Strongly  More  Preferable  than  CCAS 


Inconsistency  Ratio  =  0.047 


RESULTS: 


PA 

CPR-EZ 

CCAS 

CAPPS 


0.527 

0.041 

0.082 

0.349 

1.000 
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With  Respect  to  REGRESS  (0.406): 

PA  is  Strongly  More  Preferable  than  CPR-EZ 
PA  is  Strongly  More  Preferable  than  CCAS 
PA  is  Strongly  More  Preferable  than  CAPPS 

CPR-EZ  is  Equally  as  Preferable  as  CCAS 
CPR-EZ  is  Equally  as  Preferable  as  CAPPS 

CCAS  is  Equally  as  Preferable  as  CAPPS 


Inconsistency  Ratio  =  0.000 


results: 


PA 

0.625 

CPR-EZ 

0.125 

CCAS 

0.125 

CAPPS 

0.125 

1.000 


With  Respect  to  EAC  (0.134): 

PA  is  Very  Strongly  More  Preferable  than  CPR-EZ 

PA  is  Strongly  More  Preferable  than  CCAS 

PA  is  Strong  to  Very  Strongly  More  Preferable  than  CAPPS 

CCAS  is  Moderate  to  Strongly  More  Preferable  than  CPR-EZ 
CAPPS  is  Moderately  More  Preferable  than  CPR-EZ 

CCAS  is  Equal  to  Moderately  More  Preferable  than  CAPPS 


Inconsistency  Ratio  =  0.057 


results: 


PA 

CPR-EZ 

CCAS 

CAPPS 


0.641 

0.056 

0.186 

0.117 

1.000 
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With  Respect  to  NARRATIV  (0.054): 

PA  is  Very  Strong  to  Extremely  More  Preferable  than  CPR-EZ 
PA  is  Extremely  More  Preferable  than  CCAS 
PA  is  Extremely  More  Preferable  than  CAPPS 

CPR-EZ  is  Moderately  More  Preferable  than  CCAS 
CPR-EZ  is  Moderately  More  Preferable  than  CAPPS 

CCAS  is  Equally  as  Preferable  as  CAPPs 


Inconsistency  Ratio  =  0.045 


RESULTS: 


PA 

0.730 

CPR-EZ 

0.147 

CCAS 

0.061 

CAPPS 

0.061 

1.000 

MOD/PRNT  Category  (0.119): 

With  Respect  to  PRTPLPRN  (0.517): 

PA  is  Equally  as  Preferable  as  CPR-EZ 
PA  is  Equally  as  Preferable  as  CCAS 
PA  is  Equally  as  Preferable  as  CAPPS 

CPR-EZ  is  Equally  as  Preferable  as  CCAS 

CPR-EZ  is  Equal  to  Moderately  More  Preferable  than  CAPPS 

CCAS  is  Equal  to  Moderately  More  Preferable  than  CAPPS 


Inconsistency  Ratio  =  0.022 


results: 


PA 

0.246 

CPR-EZ 

0.289 

CCAS 

0.289 

CAPPS 

0-175 

1.000 
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With  Respect  to  PRTGRDTA  (0.359): 

PA  is  Moderately  More  Preferable  than  CPR-EZ 
CCAS  is  Equal  to  Moderately  More  Preferable  than  PA 
PA  is  Very  Strongly  More  Preferable  than  CAPPS 

CCAS  is  Moderate  to  Strongly  More  Preferable  than  CPR-EZ 
CPR-EZ  is  Strongly  More  Preferable  than  CAPPS 

CCAS  is  Very  Strong  to  Extremely  More  Preferable  than  CAPPS 


Inconsistency  Ratio  =  0.044 


RESULTS : 


PA 

CPR-EZ 

CCAS 

CAPPS 


0.315 

0.143 

0.498 

0.044 

1.000 


With  Respect  to  MODIFY  (0.124): 

PA  is  Strong  to  Very  Strongly  More  Preferable  than  CPR-EZ 
PA  is  Extremely  More  Preferable  than  CCAS 
PA  is  Extremely  More  Preferable  than  CAPPS 

CPR-EZ  is  Strong  to  Very  Strongly  More  Preferable  than  CCAS 
CPR-EZ  is  Strong  to  Very  Strongly  More  Preferable  than  CAPPS 

CCAS  is  Equally  as  Preferable  as  CAPPS 


Inconsistency  Ratio  =  0.092 


RESULTS : 


PA 

CPR-EZ 

CCAS 

CAPPS 


0.681 

0.219 

0.050 

0.050 

1.000 
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PC  Category  (0.094): 

PA  is  Equally  as  Preferable  as  CPR-EZ 

PA  is  Equally  as  Preferable  as  CCAS 

PA  is  Equally  as  Preferable  as  CAPPS 

CPR~EZ  is  Equally  as  Preferable  as  CCAS 
CPR-EZ  is  Equally  as  Preferable  as  CAPPS 

CCAS  is  Equally  as  Preferable  as  CAPPS 


Inconsistency  Ratio  =  0.000 


RESULTS : 


PA 

CPR-EZ 

CCAS 

CAPPS 


0.250 

0.250 

0.250 

0.250 

1.000 


INTERACT  Category  (0.058): 

PA  is  Very  Strongly  More  Preferable  than  CPR-EZ 
PA  is  Equal  to  Moderately  More  Preferable  than  CCAS 
PA  is  Moderate  to  Strongly  More  Preferable  than  CAPPS 

CCAS  is  Strong  to  Very  Strongly  More  Preferable  than  CPR-EZ 
CAPPS  is  Moderately  More  Preferable  than  CPR-EZ 

CCAS  is  Moderate  to  Strongly  More  Preferable  than  CAPPS 


Inconsistency  Ratio  =  0.036 


RESULTS: 


PA 

CPR-EZ 

CCAS 

CAPPS 


0.491 

0.054 

0.336 

0.119 

1.000 
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DOCUMENT  Category  (0.036): 

PA  is  Extremely  More  Preferable  than  CPR-EZ 

PA  is  Very  Strongly  More  Preferable  than  CCAS 

PA  is  Strong  to  Very  Strongly  More  Preferable  than  CAPPS 

CCAS  is  Strongly  More  Preferable  than  CPR-EZ 
CAPPS  is  Strongly  More  Preferable  than  CPR-EZ 

CAPPS  is  Equal  to  Moderately  More  Preferable  than  CCAS 


Inconsistency  Ratio  =  0.097 

RESULTS : 


PA 

0.674 

CPR-EZ 

0.040 

CCAS 

0.117 

CAPPS 

0.168 

1.000 

HELP  Category  (0.023): 

PA  l  Strongly  More  Preferable  than  CPR-EZ 
PA  is  Very  Strongly  More  Preferable  than  CCAS 
CAPPS  is  Moderately  More  Preferable  than  PA 

CPR-EZ  is  Moderate  to  Strongly  More  Preferable  than  CCAS 
CAPPS  is  Very  Strongly  More  Preferable  than  CPR-EZ 

CAPPS  is  Extremely  More  Preferable  than  CCAS 


Inconsistency  Ratio  =  0.087 


RESULTS: 


PA 

CPR-EZ 

CCAS 

CAPPS 


0.289 

0.093 

0.040 

0.579 

1.000 
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THE  FINAL  RANKINGS  OF  THE  FOUR  CPR  ANALYSIS  SOFTWARE  PROGRAMS  USING 


THE  EXPERT  CHOICE  VERBAL  MODE  (See  Table  11)  are  listed  below: 


CPR  Analysis 

Software  Program  Ranking 


PA  0.479 
CAPPS  0.209 
CCAS  0.176 
CPR-EZ  0.136 

1.000 


Expert  Choice  Numerical  Mode 

As  stated  previously,  the  first  step  in  using  Expert  Choice  to 
evaluate  the  CPR  analysis  software  packages  was  to  evaluate  the  criteria 
against  each  other  with  regard  to  order  of  importance.  The  same  approach 
was  taken  as  in  the  verbal  mode.  That  is,  first  the  seven  general 
categories  (FRIENDLY,  TOOLS,  M0D/PRNT,  PC,  INTERACT,  DOCUMENT  and  HELP) 
listed  in  Table  8  were  compared.  Then  the  subcriteria  in  the  FRIENDLY, 
TOOLS  and  MOD/PRNT  categories  were  compared.  Finally,  the  four  CPR 
analysis  software  packages  were  evaluated  against  each  other  with  respect 
to  each  of  the  fourteen  criteria. 

The  seven  general  categories  and  the  subcriteria  in  the  FRIENDLY, 
TOOLS  and  MOD/PRNT  categories  were  evaluated  based  on  the  weights  in 
Table  8  for  a  numerical  analysis.  The  overall  weights  of  the  FRIENDLY, 
TOOLS  and  MOD/PRNT  categories  were  determined  by  averaging  the  weights  of 
the  respective  subcriteria  in  each  category.  The  weights  used  in 
evaluating  the  seven  general  categories  as  well  as  the  subcriteria  in  the 
FRIENDLY,  TOOLS  and  MOD/PRNT  categories  were  listed  in  Table  12.  Table 
12  is  reconstructed  below.  The  four  CPR  analysis  software  packages  were 
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evaluated  using  the  same  comparisons  conducted  in  the  verbal  analysis 
mode. 


As  stated  previously,  the  comparison  was  conducted  in  three  basic 
steps.  The  criteria  evaluated  against  each  other  first  were  the 
following  seven  general  categories:  PRIENDLY,  TOOLS,  MOD/PRNT,  PC, 
INTERACT,  DOCUMENT  and  HELP. 
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PIRST  STEP  (With  Respect  to  GOAL): 

FRIENDLY/TOOLS  =  18.333/16.24975  =  1.128202 

FRIENDLY  is  1.1  times  (EQUALLY)  More  Important  than  TOOLS 

FRIENDLY/MOD/ PRNT  =  18.333/12.888667  =  1.4224124 

FRIENDLY  is  1.4  times  (EQUALLY)  More  Important  than  MOD/PRNT 

FRIENDLY/PC  =  18.333/12.333  =  1.4864996 

FRIENDLY  is  1.5  times  (EQUAL  TO  MODERATELY)  More  Important 

than  PC 

FRIENDLY/ INTERACT  =  18.333/11.333  =  1.6176652 

FRIENDLY  is  1.6  times  (EQUAL  TO  MODERATELY)  More  Important 

than  INTERACT 

FRIENDLY/DOCUMENT  =  18.333/9.333  =  1.9643202 

FRIENDLY  is  2.0  times  (EQUAL  TO  MODERATELY)  More  Important 

than  DOCUMENT 

PRIENDLY/HELP  =  18.333/8.667  =  2.1152648 

FRIENDLY  is  2.1  times  (EQUAL  TO  MODERATELY)  More  Important 

than  HELP 

TOOLS /MOD/PRNT  =  16.24975/12.888667  =  1.2607782 

TOOLS  is  1.3  times  (EQUALLY)  More  Important  than  MOD/PRNT 

TOOLS/PC  =  16.24975/12.333  =  1.3175829 

TOOLS  is  1.3  times  (EQUALLY)  More  Important  than  PC 

TOOLS /INTERACT  =  16.24975/11.333  =  1.4338436 

TOOLS  is  1.4  times  (EQUALLY)  More  Important  than  INTERACT 

TOOLS/DOCUMENT  =  16.24975/9.333  =  1.7411068 

TOOLS  is  1.7  times  (EQUAL  TO  MODERATELY)  More  Important 
than  DOCUMENT 

TOOLS/HELP  =  16.24975/8.667  =  1.874899 

TOOLS  is  1.9  times  (EQUAL  TO  MODERATELY)  More  Important 
than  HELP 

MOD/PRNT/PC  =  12.888667/12.333  =  1.0450553 

MOD/PRNT  is  1.0  times  (EQUALLY)  as  Important  as  PC 

MOD/PRNT/ INTERACT  =  12.888667/11.333  =  1.1372688 

MOD/PRNT  is  1.1  times  (EQUALL)  More  Important  than  INTERACT 

MOD/PRNT/DOCUMENT  =  12.888667/9.333  =  1.3809779 

MOD/PRNT  is  1.4  times  (EQUALLY)  More  Important  than  DOCUMENT 
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MOD/PRNT/HELP  =  12.888667/8.667  =  1.4870967 

MOD/PRNT  is  1.5  times  (EQUAL  TO  MODERATELY)  More  Important 

than  HELP 

PC/INTERACT  =  12.333/11.333  =  1.0882379 

PC  is  1.1  times  (EQUALLY)  More  Important  than  INTERACT 

PC/DOCUMENT  =  12.333/9.333  =  1.3214401 

PC  is  1.3  times  (EQUALLY)  More  Important  than  DOCUMENT 

PC/HELP  =  12.333/8.667  =  1.4229837 

PC  is  1.4  times  (EQUALLY)  More  Important  than  HELP 

INTERACT/DOCUMENT  =  11.333/9.333  =  1.2142934 

INTERACT  is  1.2  times  (EQUALLY)  More  Important  than  DOCUMENT 

INTERACT/HELP  =  11.333/8.667  =  1.3076036 

INTERACT  is  1.3  times  (EQUALLY)  More  Important  than  HELP 

DOCUMENT/HELP  =  9.333/8.667  =  1.0768432 

DOCUMENT  is  1.1  times  (EQUALLY)  More  Important  than  HELP 


Inconsistency  Ratio  =  0.000 


RESULTS : 


FRIENDLY 

0.205 

TOOLS 

0.183 

MOD/PRNT 

0.143 

PC 

0.139 

INTERACT 

0.128 

DOCUMENT 

0.105 

HELP 

0.097 

1.000 


The  second  basic  step  consisted  of  evaluating  the  subcriteria  in 
each  of  the  following  three  general  categories  against  each  other: 
FRIENDLY,  TOOLS  and  MOD/PRNT. 

SECOND  STEP: 

With  Respect  to  FRIENDLY  (0.205): 

SCREENS /INSTALL  =  21/18.333  =  1.1454754 

SCREENS  is  1.1  times  (EQUALLY)  More  Important  than  INSTALL 
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SCREENS /PRODTVTY  =  21/15.667  =  1.340397 

SCREENS  is  1.3  times  (EQUALLY)  More  Important  than  PRODTVTY 

INSTALL/ PRODTVTY  =  18.333/15.667  =  1.1701666 

INSTALL  is  1.2  times  (EQUALLY)  More  Important  than  PRODTVTY 


RESULTS : 


SCREENS  0.373 
INSTALL  0.341 
PRODTVTY  0.286 

1.000 


With  Respect  to  TOOLS  (0.183): 

VARIANCE/REGRESS  =  19.333/19.333  =1.0 

VARIANCE  is  1.0  times  (EQUALLY)  as  Important  as  REGRESS 

VARIANCE/EAC  =  19.333/15.333  =  1.2608752 

VARIANCE  is  1.3  times  (EQUALLY)  More  Important  than  EAC 

VARIANCE/NARRATIV  =  19.333/11.0  =  1.7575455 

VARIANCE  is  1.8  times  (EQUAL  TO  MODERATELY)  More  Important 

than  NARRATIV 

REGRESS /EAC  =  19.333/15.333  =  1.2608752 

REGRESS  is  1.3  times  (EQUALLY)  More  Important  than  EAC 

REGRESS /NARRATIV  =  19.333/11.0  =  1.7575 

REGRESS  is  1.8  times  (EQUAL  TO  MODERATELY)  More  Important 

than  NARRATIV 

EAC/NARRATIV  =  15.333/11.0  =  1.3939091 

EAC  is  1.4  times  (EQUALLY)  More  Important  than  NARRATIV 


RESULTS : 


VARIANCE  0.301 
REGRESS  0.301 
EAC  0.232 
NARRATIV  0.167 

1.001 
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With  Respect  to  MOD/PRNT  (0.143): 

PRTGRDTA/MODIFY  =  13.0/12.333  =  1.0540825 

PRTGRDTA  is  1.1  times  (EQUALLY)  More  Important  than  MODIFY 

PRTPLPRN/MODIFY  =  13.333/12.333  =  1.0810833 

PRTPLPRN  is  1.1  times  (EQUALLY)  More  Important  than  MODIFY 

PRTPLPRN / PRTGRDTA  =  13.333/13.0  =  1.0256154 

PRTPLPRN  is  1.0  times  (EQUALLY)  as  Important  as  PRTGRDTA 


RESULTS: 


MODIFY  0.313 
PRTGRDTA  0.344 
PRTPLPRN  0.344 

1.001 


The  third  step  consisted  of  evaluating  the  four  CPR  analysis 
software  packages  against  each  other  with  respect  to  each  of  the  fourteen 
criteria. 


THIRD  STEP: 

The  CPR  analysis  software  packages  were  evaluated  using  the  same 
comparisons  conducted  in  the  verbal  analysis  mode  (See  THIRD  STEP  of 
"Expert  Choice  Verbal  Mode").  THE  FINAL  RANKINGS  0'  THE  FOUR  CPR 
ANALYSIS  SOFTWARE  PROGRAMS  USING  THE  EXPERT  CHOICE  NUMERICAL  MODE  are 
listed  below  (See  Table  13). 


CPR  Analysis 

Software  Program  Ranking 

PA  0.460 

CAPPS  0.213 

CCAS  0.189 

CPR-EZ  0.138 

1.000 
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